Artificial Intelligence is excerpted from The Line: AI and the Future of Personhood by James Boyle. The MIT Press Published: 2024 (Transcript, 2022). You can read the whole book for free here.

There is no security . . . against the ultimate development of mechanical consciousness, in the fact of machines possessing little consciousness now. A mollusc has not much consciousness. Reflect upon the extraordinary advance which machines have made during the last few hundred years, and note how slowly the animal and vegetable kingdoms are advancing. The more highly organised machines are creatures not so much of yesterday, as of the last five minutes, so to speak, in comparison with past time. Assume for the sake of argument that conscious beings have existed for some twenty million years: see what strides machines have made in the last thousand! May not the world last twenty million years longer? If so, what will they not in the end become? Is it not safer to nip the mischief in the bud and to forbid them further progress?

THE BUTLERIAN CHALLENGE

That passage was written in 1872. Samuel Butler, the anti-Victorian iconoclast whose novel The Way of All Flesh is one of the most searing critiques of the hypocrisies of his time, wrote a book 150 years ago that muses extensively on the possibility of machine consciousness.

Erewhon is a hard book to explain. The title is (nearly) "nowhere" backward—the same thing that "utopia" means in Greek. Erewhon is an imaginary country and it is no utopia. Instead, it is a fun-house mirror in which attentive readers could see Victorian society, and perhaps our society, reflected, reversed. The Erewhonians treat crime the way we do sickness and sickness the way we do crime, imprisoning people for being ill and relying on polite hypocrisies about criminality to excuse their own behavior. How nice it would be to say, "I'd love to come to your party, but I feel some shoplifting coming on." They punish people for having bad fortune. Arguably, so do we, and that is Butler's point. Their musical banks parallel Victorian churches. The currency the musical banks traffic in is honored piously as the true wealth but hypocritically ignored in practice, where real money is what counts. Their universities are "colleges of unreason," teaching abstruse and archaic doctrines but failing to inspire true critical thinking. As an academic myself, I'll leave that one alone. Their society even bans the killing of animals and the eating of meat, leading repressed carnivores to feel shame and often contract disease when they finally turn to the black market to gratify their illicit desires. It is a nice parallel to Victorian society's sexual repression, coupled with its enormous, brutal sex trade. To put it mildly, little in the book is as it seems.

Unwary readers who encounter the two chapters about machine consciousness out of context can be excused for taking them at face value. Was Butler seriously exploring the possibility of machine consciousness? Was he so worried about rogue AI that he even proposed a ban on mechanical progress? Certainly, some people have read him that way. If you know Frank Herbert's classic science fiction novel Dune, you have read about the "Butlerian Jihad" that banned machine intelligences in a distant future. The original Butler would have been amused by that nickname, I think. But just as the musical banks, the courts of illness, and the colleges of unreason are not what they appear to be, the discussion of machine intelligence was mainly supposed to be an allegory for another issue: his era's passionate debate over the scientific truth and theological implications of biological evolution.

Just what Butler was trying to say is a matter of some dispute. He himself seems either to have been deliberately ambiguous about it or to have changed his position. Some say he was criticizing evolution, claiming that the same arguments put forward for the gradually increasing complexity of biological beings driven by natural selection would imply that machines could develop consciousness in similar ways. If so, the reductio ad absurdum is no longer so absurdum. Others say he was using the same form of argument to parody evolution's critics, and their relentless attempts to suppress, deny, stigmatize, and, if necessary, forbid evolution's teachings. That one has an unpleasantly modern ring, too.

Butler could have been using machine consciousness as a critical allegory of evolution or an allegory against evolution's critics. Either way, a Victorian-era satirical dystopia accurately predicts our contemporary debates about thinking machines. It is as if Gulliver's Travels turned out to be a Yelp review of Lilliput as a tourist destination. ("Watch out for the little guys with the ropes! Would rate this place zero stars if I could!") There is a lesson in that. Whether or not he was serious, Butler was right that the same arguments that support biological evolution at least suggest the possibility of machine consciousness. Indeed, as we will see, one possible method of machine learning relies explicitly on an evolutionary mechanism, though the "selfish genes" are algorithms and neural networks running on computers, competing for successful reproduction into the next generation. My imaginary Hal used just such a technique. But Butler is also right that the denunciations of evolution, the explanations of why it is scientifically impossible, will parallel relatively precisely some of the denunciations of AI consciousness and the philosophical explanations that it is impossible. It is worth remembering that the critics were wrong about evolution.

More generally, Butler's work is a good starting place for our discussion for three reasons. First, Butler sees the fragility of the line, its contingent quality. Over the last 40 years, scientists such as the primatologist Frans de Waal have posed skeptical challenges to the idea of a firm, qualitative distinction between humans and nonhuman animals, finding examples of tool use, language, and so on in the animal world. But more than a hundred years earlier, Butler was pointing out that the lines between human and animal and human and machine are fuzzier than we might like to imagine. In fact, in words that seem deliberately provocative, Butler challenges both the machine-animal distinction and the idea of qualitatively distinct human consciousness:

Where does consciousness begin, and where end? Who can draw the line? Who can draw any line? Is not everything interwoven with everything? Is not machinery linked with animal life in an infinite variety of ways? The shell of a hen's egg is made of a delicate white ware and is a machine as much as an egg-cup is: the shell is a device for holding the egg, as much as the egg-cup for holding the shell: both are phases of the same function; the hen makes the shell in her inside, but it is pure pottery. She makes her nest outside of herself for convenience' [sic] sake, but the nest is not more of a machine than the egg-shell is. A "machine" is only a "device."

Having taken a shot at the firmness of the machine-animal distinction, Butler turns to self-awareness. Probably tongue-in-cheek, but no less enlightening for all that, Butler then muses on the consciousness of the humble potato:

Even a potato in a dark cellar has a certain low cunning about him which serves him in excellent stead. He knows perfectly well what he wants and how to get it. He sees the light coming from the cellar window and sends his shoots crawling straight thereto: they will crawl along the floor and up the wall and out at the cellar window; . . . we can imagine him saying, "I will have a tuber here and a tuber there, and I will suck whatsoever advantage I can from all my surroundings. This neighbour I will overshadow, and that I will undermine; and what I can do shall be the limit of what I will do. He that is stronger and better placed than I, shall overcome me and him that is weaker I will overcome." The potato says these things by doing them, which is the best of languages. What is consciousness if this is not consciousness?... We find it difficult to sympathise with the emotions of a potato; so we do with those of an oyster... Since... they do not annoy us by any expression of pain we call them emotionless; and so qua mankind they are; but mankind is not everybody.

Now Butler has the attention not just of the Dune reader but the vegetarian, who suddenly realizes that even vegetables might not be fair game. Butler's tongue-in-cheek ode to the possibilities of mind in everything from a steam engine to a potato actually fits into a once-maligned theory of consciousness now enjoying a modest revival. Panpsychism, which dates back to ancient Greece, claims that mentality or mind is everywhere. It pervades material objects as well as living beings. Adherents run the gamut from mystics to scientists who believe we overstate the differences between animate and inanimate. To be fair, most contemporary panpsychists believe that consciousness reaches its fully developed form only in beings of sufficient complexity, but the potential is there in the humblest of things.

The second reason why Butler is a good starting point for any discussion of the possibility of machine consciousness is even more basic. More than a century ago, he saw that any account of human consciousness that admits it comes from physical interactions in the brain and the nervous system will find it hard to explain why other sets of physical interactions, based on nonorganic processes, cannot produce consciousness. To put it another way, if we deny consciousness to machines because no true consciousness can come from such a programmed, materialist origin, can we call ourselves conscious? Here, again, is Butler from 1887: "[T]he theory that living beings are conscious machines, can be fought as much and just as little as the theory that machines are unconscious living beings; everything that goes to prove either of these propositions goes just as well to prove the other also." Seventy years later, Turing would use a similar argument in favor of the Imitation Game, or Turing Test for machine intelligence. If we cannot tell whether an entity is machine or human, even after extensive interaction, who are we to deny another entity consciousness? What ground do we have to stand on?

Finally, Butler's writing gives me, at least, a timescale for the debate. "The Book of the Machines" was written 150 years ago. The most complex machines around Butler were steam engines, industrial looms, and mechanical calculators. Perhaps one could add the partially completed Babbage Difference Engine, beloved of steampunk science fiction readers and computer science historians. Yet in that context, unimaginably primitive in our terms, he could still see that in the grand sweep of time, "[t]he more highly organised machines are creatures not so much of yesterday, as of the last five minutes." In other words, he could warn us—with our Siris and ChatGPTs and our deep learning, convolutional neural nets massaging big data—that the timescale of these advances is so short historically, and the pace so rapid, that we should doubt our ability to extrapolate confidently in either direction about the journey's final destination. That fact should discourage hubris both in those who are skeptical Artificial Intelligence will ever be developed, and those who are confident that it will arrive in some specific anticipated format and revolutionize the world in the very near future. Hubris, however, appears to be an endlessly renewable resource.

HUBRIS AND HUMILITY IN AI

The history of AI is a history of overconfident predictions. In August 1955 a group of academic luminaries submitted a grant proposal to the Rockefeller Foundation for a summer workshop on AI. The document is famous partly for its historical importance—and it is a grant proposal. Every time I read it, I find myself imagining equivalent texts from other historical moments. ("Executive Summary: Goal: to escape from slavery under Pharaoh. Needs: Method of parting the Red Sea. Also, snacks.") But the document is also famous for its ambition—beginning a dialectic in AI research between wildly optimistic claims and pessimistic laments of difficulty that continues to this day. Note the goals:

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

For a summer. Progress was not quite as fast as they imagined. Nevertheless, ten years later, giants in the field such as Marvin Minsky and Herbert Simon were predicting General-Purpose Artificial Intelligence or "machines . . . capable . . . of doing any work a man can do" by the 1980s.8 Huge strides have been made in aspects of artificial intelligence— machine-aided translation, facial recognition, autonomous locomotion, expert systems, and so on. But General AI—an intelligence that exhibits all the qualities of human intelligence and capability—has remained out of reach. Indeed, because the payoff from these more limited subsystems— which today power everything from Google Translate and image recognition to the recommendations of your streaming service—is so rich, some researchers have argued that the goal of General AI was a snare and a delusion. What was needed instead, they claimed, was a set of ever more powerful subspecialties—expert systems capable of performing discrete tasks extremely well but without the larger goal of achieving consciousness or passing the Turing Test. There might be "machines capable of doing any work a man can do," but they would be multiple different machines, with no ghost in the gears, no claim to a holistic consciousness.

It is worth noting that, under some definitions, that might be enough to be hailed as Artificial General Intelligence. For example, Metaculus, a site that solicits and aggregates predictions of future events, has as its criteria for high-level General AI that it has to be able to pass a two-hour adversarial Turing Test featuring text and images, assemble a complex model car, and perform well on tests assessing a number of other capabilities. The focus is on capabilities. In other words, if we could have a machine that did all of the things humans can do, from composing a sonnet to conversing fluently, from changing a lightbulb to piloting a plane, that would be enough. The development of such a multitalented machine would certainly transform our economy and society, but my interest is in AI personhood and potential consciousness; being an extremely competent collection of expert systems is not automatically enough. Beyond those skills, I am asking the question of whether there is some consciousness, some set of morally salient capabilities, that would cause us to see the machine as a moral actor whose personhood should be recognized.

Despite the history of overconfidence and of setbacks, arguments that General AI will appear in the near future have not ended. Indeed, if anything, the optimistic claims have become even more far-reaching. Thirty years ago the buzzword among the most fervent AI optimists was the Singularity, a sort of technological liftoff point in which a combination of scientific and technical breakthroughs lead to an explosion of self­improving Artificial Intelligence coupled to a vastly improved ability to manipulate both our bodies and the external world through nanotechnology and genetic engineering. Writers such as Vernor Vinge and Ray Kurzweil used the term Singularity to refer to the point where, because of exponential technological growth, the graph of technological progress will go vertical or at least be impossible to predict using current tools. Assuming explosive and imminent advances in AI, they believed that we would soon have improvements not in technology alone, but in the intelligence that will create new technology. Intelligence itself will be transformed. Once we have built machines smarter than ourselves— machines capable of building machines smarter than themselves—we will, by definition, be unable to predict the line that progress will take. Vinge, whose 1993 article initiated the focus on an AI Singularity, was pessimistic about what might result. Why should we assume that an intelligence vastly greater than our own would treat us any better than we treat chimpanzees? Kurzweil, by contrast, generally saw the Singularity leading us into a glorious world of posthuman immortality.

Kurzweil's view seemed to resonate more in frothy, popular science discussions, but, in recent years, an alternative to Kurzweil's view has developed, one that hearkens back to Vinge's original caution. This perspective, associated with researchers such as Eliezer Yudkowsky and Nick Bostrom, shares with Kurzweil the intuition that Artificial General Intelligence may arrive much sooner than many of us expect. It differs in that the consequences it foresees are by no means as benign. In Yudkowsky's words, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." Far from bringing us immortality and a peaceful and insanely productive, ecologically sustainable world, Yudkowsky and his fellow skeptics argue that the Singularity could bring global devastation and even human extinction.

The term "singularity" is actually drawn from a memorial tribute given by Stanisiaw Ulam to the famous mathematician and information theorist John von Neumann. It is normally quoted in an abbreviated form that suggests von Neumann's eminence can be enlisted in support of the optimistic Singularity vision. Read in full and in context, however, the original quotation uses the term "singularity" to refer to a different and less positive set of possibilities than Kurzweil's image. Ulam says of von Neumann:

Quite aware that the criteria of value in mathematical work are, to some extent, purely aesthetic, he once expressed an apprehension that the values put on abstract scientific achievement in our present civilization might diminish: "The interests of humanity may change, the present curiosities in science may cease, and entirely different things may occupy the human mind in the future." One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Far from racing with delirious optimism into a technologically transformed future, I read von Neumann, and perhaps Ulam, to have apprehensions about the "changes in the mode of human life" in a future that they could not predict and in which "human affairs, as we know them, could not continue." This is hardly the full-throated endorsement of the optimistic Singularity. In fact, it sounds a Burkean note of caution that would later be echoed in Bostrom and Yudkowsky's darker visions of how AI might transform or destroy our world.

On the other hand, von Neumann is putting forward two premises central to the contemporary usage of the term. First, technological progress—or at least technological progress in some fields—is exponential, not linear. (But for how long?) Second, while the first few stages of an exponential graph are not that different from a linear one, the line on the graph quickly goes almost vertical. This will lead those who are assuming more linear growth, or who are standing on the flatter part of the time curve, to dramatically overestimate how long technological developments will take to achieve. It will also rapidly put the future out of sight from where we are, thus rendering it impossible to predict. Strikingly, despite this fact, some of the proponents of the Singularity do prophesy with apparent confidence about what will transpire after it occurs. Kurzweil imagines a posthuman, technologically enabled immortality, for example.

To the uninitiated, the future painted in Kurzweil's 2005 The Singularity Is Near sounds like a delightfully wacky fantasy, a high-tech version of the rapture in which our posthuman bodies rise up to an endless virtual reality in the cloud, run by benign intelligences that have long ago transcended our limits. A "version" of the rapture? That is the rapture. No wonder the more enthusiastic odes to the Singularity have a religious, chiliastic feel to them. Sometimes, that impression can get in the way of a careful assessment of the specific claims being made about AI that, while overly optimistic, are based on thought-provoking premises.

If technological change (e.g., the doubling of computer chip capacity every months to two years that is known as Moore's law) could continue on an exponential curve, then a dramatically different future will arrive far sooner than we expect. That is Kurzweil's central point, as it was von Neumann's. But many scientists warn that we are rapidly approaching the physical limits of science in making transistors smaller. What's more, some have argued that, at our current levels of technology, cost- benefit analysis will no longer support the titanic investments required to continue to meet that benchmark. Moore's law may have ceased to be true already. The exponential graph may flatten out, whether it is flattened by physics or balance sheets or both.

To be fair to those who believe in a short timeline to General AI, they generally do not predict a single, invariant, exponential curve but rather a stacked series of S-curves in which a particular technology starts off slowly, hits an exponential period of innovation, flattens off, and is in turn replaced by a new technology that goes through the same stages. One way for this trend to continue in the realm of computer architecture would require us to predict, for example, that current chip designs would be overtaken by a new paradigm—quantum computing, say, which would exploit the physics of the quanta such as the entanglement of quantum particles, Einstein's "spooky action at a distance." Of course, the dramatic advance does not have to be quantum computing. Perhaps Richard Feynman was right and there is still room at the bottom, in the nanoscale, using technologies and heat dissipation methods we only dimly understand now. Or perhaps some combination of biological computing and machine computing will open the next frontier. Perhaps the transformation will not primarily be to the hardware at all but rather in the software, with new techniques of machine learning producing quantum leaps in performance. Regardless of the specific technique, the large claim is that we will continue to find new revolutionary technologies that will enable yet another S-curve in computer capacity. Yet how can we confidently predict such paradigm shifts in technology? By definition, they are outside of our current technological frame of reference.

The speed of technological transformation will be particularly hard to predict if we are talking about multiple technologies, sometimes accelerating on exponential curves, having unexpected synchronistic effects on each other. Take the evolution of computer networks from 1990 to 2005, for example. Most of the basic technological components of the internet were there in the 1980s. Versions of the internet itself—a distributed packet-switched system—date back to the 1950s. But during this period of time, those things suddenly came together to form the World Wide Web, to revolutionize our communications, our media, and our global commerce.

We can debate what addition supersaturated the solution and precipitated the crystal of transformative innovation—Tim Berners-Lee's architecture of HTML and the World Wide Web? The price, speed, and memory frontier that PCs hit in the early 1990s? The unused bandwidth available on cable networks' fiber backbones due to networks and "rights of way" property regimes created for an entirely different purpose? More likely, it is all of the above. Without any single great breakthrough, the world was suddenly dramatically different. The worldwide internet went from being a science fiction trope that was never going to exist ("flying cars!") to a reality in about five years. It became the reality—an unquestioned feature of our world like gravity and oxygen—in a mere 15. People who had predicted for decades that computers and networks would transform society, and faced entirely justified heckling when the promised revolution failed to appear, were wrong, wrong, wrong until they were suddenly and shockingly right. In 15 years, the world changed dramatically, without warning and without some eureka discovery that might have been thought necessary to precipitate the transformation. All the technologies were well understood. The result was not. That incident is undeniably part of our past. And we think we can predict the future?

Why can this not happen with AI? I do not mean to say that it will, but confident assertions either way should be met with skepticism. Duke Law School's parking lot has some gratifyingly witty bumper stickers. One seems appropriate here. "Radical Agnostic" says the large, capitalized text. Underneath is the smaller punch line. "I don't know and you don't either!!" Perhaps this should be our motto for AI prognostication. Some may think, perhaps rightly, that I fail that test. I am going to argue that there are reasons to believe that progress is likely to be faster than many of us think. My agnosticism has a tilt. Nevertheless, I think the radical agnostic's motto is the right one.

If the internet's transformation seems too singular and unlikely to be representative, it is worth remembering that we have just lived through another example of this process of synchronistic change: the rapid proliferation of neural network systems that rely on deep learning to recognize speech in multiple languages, translate sentences, identify pictures, predict consumer desires, and so on. How did this happen? The origins of electronic neural networks can be found as far back as the 1940s and 1950s. A cluster of events had to come together to produce the leap forward of the last ten years. There were revolutionary breakthroughs in network theory and design—the software side. Continuous improvements in speed and drops in cost of hardware made those software advances suddenly have a much greater reach or potential. But wider cultural and technological transformations also played a role. Both the software and hardware showed what they could do because of an explosion of data on which they could be tested and proven.

Combine the continuously improving technologies of the individual computer—which is rapidly increasing in speed, processing power, and memory capacity while dropping in price—with a global network of other computers doing the same thing and a cloud that is almost always in reach. Put those computers in people's pockets, as smartphones. Now we have nearly seven billion nodes connecting to the cloud around the planet, each performing a host of different tasks and running many different apps, and thus an exponential increase in the rate of data generation by those rapidly proliferating devices.

Millions of people navigate using Google Maps, upload and tag photographs, dictate commands to their phones, and then correct that dictation, providing feedback to the system. The torrent of data is staggering—"big data," indeed. And in that data are patterns, patterns that artificial intelligence can "learn" to identify. Rather than programming the system with rules up-front—"this is the shape of a cat," "when a British person says 'bath' it sounds like this"—the system uses an architecture very loosely emulating the organization of neurons in the brain, arranged in sequential layers of processing. Many such systems develop through a process of trial and error, giving greater weight to the input from those layers that improve the accuracy of predictions. Once programmed with goals and parameters, and in some cases with an initial curated data set, the system can perform this process on its own, layer after layer, developing its own credit-assignment paths that lead to ever more precise identification in a process that may be partially inscrutable even to the original programmers. The system might even be given almost no guidance and simply rewarded through deep reinforcement learning when it does something its programmers think is good. This technique has consistently outperformed more structured, choreographed approaches to the problems machine intelligence must solve.

Look at the number of technological developments that come together to make this happen. It is not simply a matter of Moore's law, which skeptics rightly point out is no longer empirically accurate. Deep learning depends on dramatic changes in memory capacity, price, distributed storage, number of users, and advances in artificial intelligence theory and software. It turns those advances onto the firehose of data generated by our computer systems. And the neural network uses deep learning, rather than some formal set of preprogrammed rules, to master this torrent of data. Peter Norvig, the Director of Research at Google and a leading scholar of artificial intelligence, puts it nicely: "We decided that the best model of the world was the world."

Deep learning has been a revolutionary development. Google Translate became dramatically better literally overnight. Image- or speech- recognition software was suddenly vastly more accurate. For all of this, you have deep learning, and probably neural networks, to thank. What does this tell us about the prospects of General AI? By itself, not much at all. True, this is one type of artificial intelligence, focused on discrete tasks, but it is not General AI, let alone consciousness, unless your threshold for consciousness is "can you identify all the cute little kitty cats in this picture?" Large language models such as ChatGPT or LaMDA are such systems. Blake Lemoine, the Google engineer whose story began this book, was so convinced by LaMDA's output that he believed it had become conscious. Lemoine was incorrect: there is no ghost in that machine, merely jaw-droppingly brilliant imposture.

The story of deep learning, and of Lemoine's error, do not teach us that General AI is here, or that machine learning systems like LaMDA or Chat- GPT are going to become conscious tomorrow. Instead, they should teach us something very different: that it is very hard to forecast developments in technologies, some of which are developing at exponential rates, when it is the interaction of the rapidly changing components of the system that enables the dramatic, paradigm-shifting change. The point is that sudden and unexpected change is possible, though not inevitable, whether from exponential growth within one field or syncretic fusion among many. That suggests we might want to take seriously the arguments of those who think that Artificial General Intelligence may arrive sooner than we think, even if we are skeptical of their precise timetable, or their predictions of rapturous immortality or machine-led annihilation. We need not rely on their arguments as descriptions of what will happen and when. We can think of them as reasonable suggestions of what could happen, and why.

Perhaps an anecdote will underline that point. As I was writing these words, I saw the news that Geoffrey Hinton, a renowned pioneer in neural networks, had resigned from Google so that he could speak more freely about his concerns over AI systems. This was not exactly like Thomas Edison quitting his job because he was worried about the effects of light- bulbs but, for many in the field, it produced an equivalent level of shock. To be clear, Hinton's concerns about the breakneck pace of technological development around AI systems are broad ones. He instanced everything from the rampant production of deep fakes to the effect on the labor market, warfare, and political stability. But I was struck by one thing he said: "The idea that this stuff could actually get smarter than people—a few people believed that. . . . But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." He is not alone in this belief. Google's DeepMind is without doubt one of the most important companies in the field. Its research has been vital to current breakthroughs, including those by rival companies. The same month as Hinton's resignation, DeepMind's CEO Demis Hassabis had this to say: "The progress in the last few years has been pretty incredible, I don't see any reason why that progress is going to slow down. I think it may even accelerate. So, I think we could be just a few years, maybe within a decade, away [from human-level AI]."

ARTIFICIAL INTELLIGENCE? WHEN?

This brings us to the obvious question: Will General-Purpose, or even conscious, AI arrive at all, and if so, when? It turns out that those studying AI have radically different answers to those questions. They differ about the most promising lines of research, their difficulty, and the extent to which industry and academic research scientists will actually be focused on Artificial General Intelligence rather than on building many discrete artificial intelligence systems that make hair appointments, book your travel, or organize your photo album. But they also differ on the two axes just identified: optimism or pessimism about sustained exponential growth and optimism or pessimism about the frequency and significance of technological synchronicity—the coming together of many factors to produce a leap forward that was not predictable in advance.

These forms of optimism and pessimism are shared in the discussion of economic growth more generally, of course. Tyler Cowen's The Great Stagnation and Robert Gordon's The Rise and Fall of American Growth both provide compelling arguments against the assumption that we will continue to have the kind of robust economic growth, year after year, that characterized much of the twentieth century, though Cowen is actually more optimistic. But the AI debates present a particularly hard puzzle for prediction because we have glaring examples of remarkable, and in some cases exponential, rates of technological advance. Yet we also have repeated, humility-inducing difficulties and failures. After all, some problems that AI scientists at first thought were fairly basic (teaching a computer "common sense," for example) have proven remarkably hard to solve:

A.I. "recognizes objects, but can't explain what it sees. It can't read a textbook and understand the questions in the back of the book," said Oren Etzioni, a former University of Washington professor who oversees the Allen Institute for Artificial Intelligence. "It is devoid of common sense." Success may require years or even decades of work—if it comes at all. Others have tried to digitize common sense, and the task has always proved too large. In the mid-1980s, Doug Lenat, a former Stanford University professor, with backing from the government and several of the country's largest tech companies, started a project called Cyc. He and his team of researchers worked to codify all the simple truths that we learn as children, from "you can't be in two places at the same time" to "when drinking from a cup, hold the open end up." Thirty years later, Mr. Lenat and his team are still at work on this "common sense engine"—with no end in sight.

That skepticism could be strengthened by a series of disagreements in the field about the best methods for developing even discrete expert systems, let alone Artificial General Intelligence. Should AI be neat or scruffy? Neat approaches are based on some overarching framework such as symbolic logic, and they use that framework to solve every problem. Scruffy approaches, by contrast, opportunistically use different cognitive techniques to solve different problems so that the method for translating from one language to another might be different than the method for image recognition or playing chess, and much might consist of ad hoc, individually coded heuristics based on real-world experience. Should or will AI be rule governed, based on an enormously complex but finite set of algorithms laid down at the start by its designers? Alternatively, will it be partially autonomous, "learning" how to achieve tasks in ways that may be inscrutable to the original creators? Will it be based on advances in the logical dissection of how humans actually think or on the pursuit of rational problem-solving, regardless of how humans think? Something else altogether? If the AI optimists cannot even tell us what methods will yield General AI, then how can their optimism be sustained?

Reflecting the number of questions to be answered, surveys of AI researchers have shown considerable divergence in predictions of when General AI, or something like it, would be achieved. One notable 2016 survey used as its target population all of the researchers who published at two of the most important conferences in the field and asked, among other things, when high-l evel machine intelligence would be achieved. Their definition of such intelligence was a demanding one: "High-level machine intelligence (HLMI) is achieved when unaided machines can accomplish every task better and more cheaply than human workers." Note that this definition, like any we might choose, will have dramatic effects on the outcomes. For example, we might want to know when the first example of General AI could be achieved if we were willing to put Manhattan Project-1 evel resources into it, not when every doctor, novelist, lawyer, composer, and kindergarten teacher could be replaced by a better, and cheaper, cybernetic equivalent. Alternatively, if our concerns were with the question of when there might be some moral claim to legal personhood, we might think it irrelevant whether the AI could do brain surgery or dance ballet, just as long as we felt its consciousness shared enough with our own to warrant such a claim. The advantage of the question the researchers posed is that it looks formalizable and falsifiable, avoiding philosophical debates about whether true consciousness had been or ever could be achieved. That is also its disadvantage. Still, given both its universality of field and its price constraint—every task humans can do, in every case, done cheaper—it presents a very demanding standard.

The aggregate forecast was that there was about a 30 percent chance of achieving high-1 evel machine intelligence within about 25 years (as of 2016) and a 50 percent chance of achieving it within 45 years. The researchers reported a striking demographic split in responses: "Asian respondents expect HLMI within 30 years, whereas North Americans expect it in 74 years." Interestingly, the aggregate forecast suggested there was a 10 percent chance that it might be achieved within nine years of 2016, that is by 2025! Kurzweil's view is still an outlier, but it falls, or fell, within the bounds of the profession.

As of August 2022, Metaculus, the online prediction site, was forecasting that we would have Artificial General Intelligence by November 2041. Their criteria for Artificial General Intelligence were different than the survey above; as I mentioned before, the system had to be able to perform well on tests assessing varied skills ranging from a two-hour adversarial Turing Test, featuring text and images, to the assembly of a complex model car. By May 2023, their assessment had changed. "The Metaculus community currently expects [Artificial General Intelligence] to be unveiled in October 2031." The influential AI thinker Eliezer Yudkowsky showed equal optimism about the speed of the transformation, coupled with extreme pessimism about its results. He accepted the following bet from Bryan Caplan: "Bryan Caplan pays Eliezer $100 now, in exchange for $200 CPI-adjusted from Eliezer if the world has not been ended by nonaligned AI before 12:00am GMT on January 1st, 2030."

On the other end of the spectrum from the Singularists are skeptics who find these predictions wildly optimistic (or pessimistic, depending on what you think General AI will do when it arrives). Rodney Brooks, a former director of the MIT Computer Science and Artificial Intelligence Laboratory, and the founder of iRobot, the company that makes your Roomba, has been a frequent critic of overconfident predictions. He claims they are characterized by a pattern of fallacies. They predict consistent exponential rates of technological growth rather than a regression to the mean. They use trivial accomplishments (iPhoto recognizing all the photos of your lover's face) as evidence for the idea that qualitative transformations (General AI) are close at hand. Finally, they make firm technological projections when the timescale means that neither the technology nor the state of the world in which that technology will be deployed can accurately be predicted. Brooks pointedly rejects Kurzweil's claims, and some of his own projections put human-1 evel AI much further in the future: "It will be well over 100 years before we see this level in our machines. Maybe many hundred years." Interestingly, though, it is the optimistic time-horizon and suddenness suggested by the proponents of the Singularity that Brooks doubts, not the eventual achievement itself. Instead, he imagines a gradual process of improvement, "generation by generation by generation. The singularity will be a period, not an event." We will be driven, he thinks, "not by the imperative of the singularity itself but by the usual economic and sociological forces. Eventually, we will create truly artificial intelligences, with cognition and consciousness recognizably similar to our own."

Why is Brooks so confident, given that he is generally a skeptic of optimistic AI claims? The reason is simple. We are learning more and more about the neurological processes of the brain. What we can understand, we can hope eventually to replicate:

I, you, our family, friends, and dogs—we all are machines. We are really sophisticated machines made up of billions and billions of biomolecules that interact according to well-defined, though not completely known, rules deriving from physics and chemistry. The biomolecular interactions taking place inside our heads give rise to our intellect, our feelings, our sense of self. Accepting this hypothesis opens up a remarkable possibility. If we really are machines and if— this is a big if—we learn the rules governing our brains, then in principle there's no reason why we shouldn't be able to replicate those rules in, say, silicon and steel. I believe our creation would exhibit genuine human-level intelligence, emotions, and even consciousness.

This is not the most likely method of achieving General AI, far from it. Think of Brooks's postulate as an upper bound in AI research—one way of conceiving of the problem that indicates General AI must be achievable, if incredibly hard. We have a model of a functioning consciousness: us.

Some will believe that, by divine command, consciousness can only be created by the deity, not by human hands and minds. Perhaps there is some as-yet-undiscovered emergent property of natural biological brains that cannot be reproduced, even if replicated perfectly, either in silico or even in some biological computational device. Others believe that consciousness is, in some strange way, prior to material reality—the substrate on which the observable physical universe depends—though this still begs the question of whether machines could have the requisite consciousness. But barring a divine or technologically intractable limit— some neurological equivalent of the light-speed barrier—eventually we will be able to recreate the relevant aspects of our brains and hence our consciousness. Having done that, we might be able to transcend some of the human brain's limitations in terms of speed, memory capacity, embedded knowledge base, and networked communication of thought. Starting with a model based on a physical brain we could create ever more capable forms of general, conscious Artificial Intelligence. This is extremely unlikely to be the way we would achieve General AI. In fact, it might be the hardest and the one that would take the most time. But reconceived this way, the problem becomes a material and a soluble one. And Brooks, remember, is a skeptic.

IT'S ALL ABOUT THE HARDWARE(?)

Writers on AI agree that neither the range of predictions nor the fact that the due date keeps getting bumped forward induce confidence. In his seminal 1993 article, Vinge acknowledges this fact when making his own prediction. "I believe that the creation of greater than human intelligence will occur during the next thirty years. (Charles Platt has pointed out that AI enthusiasts have been making claims like this for the last thirty years. Just so I'm not guilty of a relative-time ambiguity, let me more specific: I'll be surprised if this event occurs before 2005 or after 2030.)" This aside became known as Platt's Law: those making predictions about General AI will place its inception date roughly 30 years in the future from the date the prediction was made.

Is there some less-subjective basis on which we could predict General AI? Are there metrics that would provide us a benchmark for progress? One answer is that we do not need to replicate the specific architecture of the brain but rather to emulate, in silicon or its successors, all of the relevant capacities and capabilities of a brain—the amount of memory it can hold, how fast it can solve problems, and so on. (Hal, the imaginary computer from the introduction, achieved sentience when the number of connections in his neural networks hit a number that approximated that of a human brain. But that was a thought experiment. There is no reason to think this is the relevant metric.) Once we have equivalent hardware, goes the theory, we only need to tweak the software, and voila, General AI! But where are we in terms of comparative capabilities? And what is the historical rate of change? In 2011, eons ago in internet time, Scientific American ran the article "Computers versus Brains":

For decades computer scientists have strived to build machines that can calculate faster than the human brain and store more information. The contraptions have won. The world's most powerful supercomputer, the K from Fujitsu, computes four times faster and holds 10 times as much data. And of course, many more bits are coursing through the Internet at any moment. Yet the Internet's servers worldwide would fill a small city, and the K sucks up enough electricity to power 10,000 homes. The incredibly efficient brain consumes less juice than a dim lightbulb and fits nicely inside our head. Biology does a lot with a little: the human genome, which grows our body and directs us through years of complex life, requires less data than a laptop operating system. Even a cat's brain smokes the newest iPad—1,000 times more data storage and a million times quicker to act on it.

All of these figures, except those claimed for the brain, which are problematic for other reasons, are now out of date, of course. The 2011 Scientific American article claims that the K supercomputer could then perform 8.2 petaflops or 8.2 quadrillion (8.2 x 1015) floating-point operations per second. That was a marked advance from earlier computers. As late as 2008, IBM's Blue Gene, the fastest supercomputer at the time, was just above 1 petaflops. By contrast, the Frontier, the fastest supercomputer as of 2023, can perform 1194 petaflops, 145 times faster than the K and 1100 times faster than the Blue Gene. From the Blue Gene to the Frontier, processing speed doubled approximately every 18 months. While this may not exactly be exponential growth, it is a startling rate of improvement. And this comparative hardware approach leads people other than proponents of the Singularity to be fairly optimistic about how soon General AI will arrive. To quote Nick Bostrom, the Oxford University professor whose book Superintelligence: Paths, Dangers, Strategies warns of the dangers rather than the promise of AI:

Hardware-wise, the brain still compares favorably with machines. Estimates vary, but perhaps the cortex performs something like 1016 or 1018 operations per second using 20 watts, which is impressive. Eventually, the limits of computation in machine substrate are of course far beyond those in biological tissue, and it shouldn't take too long to reach rough equivalence. The advance of algorithms is harder to predict, but the notion that we could have human-level AI within a small number of decades seems credible, though there is great uncertainty on both the lower and upper sides of this estimate.

Bostrom's estimate of the brain's capacity is higher than that of the Scientific American article. The authors of that piece estimated the brain could perform 2 petaflops. Bostrom seems to assume that it can perform somewhere between 10 and 1,000. Jurgen Schmidhuber, scientific director of a leading Swiss AI Lab and a machine learning pioneer, is also optimistic about the arrival of General AI. His optimism is based not just on the absolute speed of the very fastest machines but on the falling price of the average machine:

When will we have computers as capable as the brain? Soon. Every five years computing is getting roughly 10 times cheaper. Unlike Moore's Law, which says that the number of transistors per microchip doubles every 18 months (and which recently broke) this older trend has held since Konrad Zuse built the first working program-controlled computer. His machine could perform roughly one floating-point operation per second. Today, 75 years later, hardware is roughly a million billion times faster per unit price. Soon we'll have cheap devices with the raw computational power of a human brain; a few decades later, of all 10 billion human brains together, which collectively probably cannot execute more than 1030 meaningful elementary operations per second.

The Open Philanthropy Project, an effective altruism nonprofit, has funded a lot of research on the possible impact of AI. In 2020 they commissioned a report on when we might have human-level AI. The author of that report, Ajeya Cotra, found a 10 percent chance by 2031, a 50 percent chance by 2052, and an almost 80 percent chance by 2100. She used a number of methods, including the "floating-point operations in the brain" analysis we have just been discussing. She even attempted, as one benchmark, to estimate the number of floating-point operations represented by the entire history of biological evolution toward humans. It is as if we saw biological evolution as a moonshot AI project trying to achieve human consciousness and could extrapolate from that how long it would take machines affordably to replicate that evolutionary path. Cotra then adjusted the sum of all of these predictive models and the median fell on, specifically, 2052, or 32 years after the report was published. A cynic might say that Platt's Law still holds! Two years later, Cotra adjusted her median prediction to 2040 because of unexpectedly good performance on a number of benchmarks since 2020.

But what do all these numbers actually mean? A critic might say that they are fundamentally misleading. Human beings do not think in floating-point operations. You can indeed calculate 1.37 times 8.91, but I am fairly sure you don't do it in a single second, still less in a millionth or billionth of a second. Nor do we conceive of the activities of recognizing a face, realizing your marinade needs more ponzu, or writing a love poem as involving floating-point operations at all. Is using this number to compare the power of a brain and a computer like using miles per hour to quantify Shakespeare's prose? To paraphrase Norvig and Russell's book Artificial Intelligence, we do not compare the albatross and the 747 by asking how quickly each flaps its wings. They achieve flight using different techniques and, barring the attempt to replicate the brain neuron by neuron, the same will be true of an attempt at building General AI.

Are these comparisons useless, then? Despite the criticisms I just pointed out, as long as they are taken with an appropriate degree of caution, such comparisons do help illuminate something useful. Any attempt to create General AI is aided by having more capable, faster, cheaper, smaller computers, which can handle more complex sets of instructions, contain more memory, form networks more easily, and so on.

In the past, artificial intelligence researchers have found that increases in speed mean problems that were once thought to require elegant solutions may in fact be solved by brute-force approaches. For example, we might think the only way to teach a computer to play chess is by elaborately programming software rules that outline strategy and tactics. Or perhaps just to have the computer teach itself by playing millions or billions of games, generating its own rules and strategies, using a technique called deep reinforcement learning. When I interviewed Hal Abelson, a renowned computer scientist at MIT, he told me that "problems that people thought could only be solved elegantly are instead being solved by simple techniques of reinforcement learning."

One of the most powerful examples of reinforcement learning is provided by the development of DeepMind's Go-playing system. The game of Go has vastly more permutations than chess: "As simple as the rules may seem, Go is profoundly complex. There are an astonishing 10 to the power of 170 possible board configurations—more than the number of atoms in the known universe. This makes the game of Go a googol times more complex than chess." With a game this mind-numbingly complicated, it would seem that any AI would have to emulate human strategies of intuition and pattern recognition and would have to rely on the tactical heuristics polished by generations of players—or not. The researchers at Google's DeepMind project created a program called AlphaGo, which went on to beat the best human players in the world. The first version of AlphaGo was "trained by supervised learning from human expert moves, and by reinforcement learning from self-play." These techniques rely on a curated dataset and an initially supervised interaction with that dataset. That is still far less direction, far less programmed strategy, than researchers had previously believed would be necessary. Yet the results of its victorious contests with human grand masters were remarkable: "During the games, AlphaGo played a handful of highly inventive winning moves, several of which—i ncluding move 37 in game two—were so surprising they overturned hundreds of years of received wisdom, and have since been examined extensively by players of all levels. In the course of winning, AlphaGo somehow taught the world completely new knowledge about perhaps the most studied and contemplated game in history."

To find the limits of deep reinforcement learning, the researchers created a second version of the program, called AlphaGo Zero,

based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules. AlphaGo becomes its own teacher: a neural network is trained to predict AlphaGo's own move selections and also the winner of AlphaGo's games. This neural network improves the strength of the tree search, resulting in higher quality move selection and stronger selfplay in the next iteration. Starting tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning 100-0 against the previously published, champion-defeating AlphaGo.

In the words of the AlphaGo Zero team: "This technique [of reinforcement learning without human guidance] is more powerful than previous versions of AlphaGo because it is no longer constrained by the limits of human knowledge."

To be clear, AlphaGo Zero is not General AI or anything remotely close to it. It also was not achieved solely because of hardware advances; the researchers at DeepMind are justifiably proud of their astonishing accomplishment in both software and neural architecture design. Chat- GPT would not exist without those advances, which the DeepMind teams shared widely. But the increase in speed, memory, and data-handling capacity that I described earlier opens entirely new possible lines of research. Neural networks, deep learning, and reinforcement learning show that we can achieve striking results at tasks previously thought to play to human strengths, such as facial recognition or intuitive strategy games, without attempting precisely to emulate the patterns of human thought.

What does this tell us? There is no one-to-one map of human and machine capability; at least at the moment, both the hardware and software are very different. Thus, the head-to-CPU comparisons of processing capabilities are wildly approximate at best. But if one trims away the hyperbole about operations per second, and number of neural connections, a truth remains. While we do not know what the crucial dimensions of hardware performance will be in eventually achieving General AI, the rate of progress on every dimension of performance suggests that Bostrom and Schmidhuber have reason for their qualified optimism. As with military strategy, greater resources mean more angles of attack, some of them previously unforeseen.

Kurzweil, of course, believes that General AI is much closer: "When will we have computers as capable as the brain? I believe computers will match and then quickly exceed human capabilities in the areas where humans are still superior today by 2029." Yudkowsky, in his pessimism, seems to believe that there is a significant danger of us achieving General AI not long after that date. From my discussions with AI researchers, I find this prediction unlikely, though some of them have become decidedly more optimistic recently. But I find equally puzzling those who claim confidently we are centuries away. The graph of technological change may not be vertical, but it is steep and punctuated by unforeseen leaps forward, sometimes driven by the synchronicity of multiple technologies unexpectedly coming together, sometimes by new approaches that harness rapidly evolving speed and big data capabilities, and sometimes by theoretical breakthroughs. At the very least, I think we can be confident of this: long before the century is out, we will have AI at a level where its consciousness is, at least, a matter on which well-informed people can, and will, reasonably disagree. The controversy will be live. Indeed, some would argue we are already there. And that is all I need.

EVEN IF IT WORKS, IS IT CONSCIOUS?

If one challenge to General AI is that it is impossible, or will take hundreds of years to achieve, a second and more fundamental challenge goes to ontology rather than technology: the nature of being, not the likelihood of working. Even if a computer-based Artificial Intelligence could do anything a human could do, would we think it was alive, aware, and thus perhaps a person? After all, it is just a machine. It is doing only what it has been programmed to do. It might replicate our responses with perfect fidelity, but would it be conscious while doing so or merely parroting lines programmed by others, like Siri "remembering" your birthday and congratulating you on it? Let us begin with Alan Turing and his critics.

In "Computing Machinery and Intelligence," Turing poses the question, "can machines think"? He then quickly suggests substituting for that question, which he calls "meaningless," another one: Can an interrogator distinguish between a human being and a machine on the basis of their typed answers to the interrogator's questions? Turing's reasons for proposing this substitution are not exactly clear. He says that it "has the advantage of drawing a fairly sharp line between the physical and the intellectual capacities of a man." He says that one alternative method of answering the question "can machines think?"—by looking at the ordinary language meaning of "machine" and "think"—is "absurd" and would lead to answering the question "by Gallup poll." He also attempts to refute a long list of objections to his alternative question—theological, mathematical, that it would not reflect true consciousness, even the assumed absence of extrasensory perception in machines. Then he concludes with disarming openness, "I have no very convincing arguments of a positive nature to support my views. If I had I should not have taken such pains to point out the fallacies in contrary views." Despite that modest disclaimer, Turing's Imitation Game has achieved considerable fame; it is now simply called the Turing Test. Should the Turing Test also be the moral or constitutional test for legal personhood? Many humans— babies, those in a coma, even those who are neurodivergent—might fail the Turing Test but are undoubtedly persons. But for those who are nonhuman, would the ability to imitate human consciousness act as the doorway to legal personhood?

The Turing Test has a lot going for it. It is relatively simple. It promises a determinate answer—a huge advantage—and one that seems designed to avoid our prejudices in favor of our own kind. The interrogator is not exactly behind a veil of ignorance, but she is attempting to deal directly with mind rather than body in a way that recalls other moments in the history of civil rights when we have been told not to focus on surface appearances. It is, as lawyers say, "formally realizable"—capable of being formulated in a test that a court or a decision maker could apply in a replicable way.

There would be questions about what the criteria of that test should be, of course. How long a conversation and under what conditions? What would be the standard of proof? What qualities would the conversation have to touch on and what qualities—i magination, humor, spirituality, morality, empathy—would it probe for? Nevertheless, at the end of the day it is something that seems more amenable to being formalized as a test than many other benchmarks of consciousness. Why? Because it seeks to convert normative judgment into statistical fact, using an "innocent" audience for greater impartiality. We do this in other areas. Want to know if a trademark presents a likelihood of confusion with another mark? The law has elaborate, albeit psychologically flawed, rules for statistically testing likely confusion with sample audiences. The Turing Test would be harder and more contentious to implement as a legal procedure, but it could look like a legal test, and that fact is significant—perhaps more than it should be. The test also presents, albeit implicitly, a challenge to our privileged position in the hierarchy of beings: If you cannot distinguish me from a human, who are you to say I am not a person?

The most famous objection to the Turing Test comes from the philosopher John Searle, who argues that effective mimicry does not in any sense imply the kind of consciousness or understanding we expect as a hallmark of thought. Searle uses the analogy of the Chinese Room—a man inside a room who does not understand Chinese but who is given an elaborate set of rules about what Chinese characters to hand back when handed characters of a particular shape. Searle's point is that those instructions might be extremely complicated, and the resulting "conversation" might seem to be a substantive one, yet in no way would the actions of the man inside the room represent consciousness or understanding in communication. It would merely be rule-following based on a characteristic (i.e., the shape of the characters) completely separate from the actual internal meaning of the words in the conversation. As a description of LaMDA, and an explanation of Blake Lemoine's mistaken attribution of personhood to it, this seems right on point.

But Searle's objection goes deeper. He is not just saying that machines programmed to pass the Turing Test are not conscious since the goal is mimicry rather than comprehension as an interior state. He is saying that machines of any kind could not be conscious. Sometimes this seems to be because, as he says, "[consciousness is a biological phenomenon like photosynthesis, digestion or mitosis." Sometimes it seems to be because he conceives of machines or artifacts as entities that are inherently operating according to a completely different set of rules than humans, programmed artifacts that have only mastered syntax as opposed to beings that also understand content and meaning, that is, semantics. In fact, those latter points seem to be definitional for him, part of the very classifications of "machine" and "programmed" rather than a contingent historical judgment about our current machines and methods of AI research. The contrasting position would be someone who believes that while we often get our artifacts to do things largely through methods of rule-based instruction—programming in the derogatory sense—from which consciousness could not spring, one could imagine different emergent properties arising from neural networks, say, evolving entirely differently in the future.

Most of the time, Searle's arguments are a combination of those last two claims: (1) consciousness is a biological property; and (2) programming cannot equal thought, no matter how precisely it mimics it.

The objection from consciousness is actually one that Turing responds to quite extensively in his original paper. He points out cogently that since we do not have direct evidence of the mental states of other human beings, we could always solipsistically posit them to be rule-following automata:

I think that most of those who support the argument from consciousness could be persuaded to abandon it rather than be forced into the solipsist position. They will then probably be willing to accept our test. I do not wish to give the impression that I think there is no mystery about consciousness. There is, for instance, something of a paradox connected with any attempt to localise it. But I do not think these mysteries necessarily need to be solved before we can answer the question with which we are concerned in this paper.

To put it another way, Turing's point is that it is no easier to prove the existence of some freestanding, nonbiologically determined entity called mind or consciousness in human beings than in computers. This is a similar point to the one Samuel Butler and B. F. Skinner made. In Skinner's words: "[T]he real question is not whether machines think but whether men do. The mystery which surrounds a thinking machine already surrounds a thinking man." Faced with the metaphysical difficulties of that move, therefore, is it not easier to look for something we can measure, namely the pragmatic evidence provided by the ability to engage in convincing unstructured communication with another human being?

In effect, Turing raises the stakes: Are you sure you aren't just a complicated Chinese Room? If you cannot prove otherwise, who are you to deny consciousness to your silicon brethren by imposing a higher burden of proof on them? In terms of constitutional law and popular debate, however, the answer to the last question is likely to be, "We're the entities who wrote the United States Constitution, that's who." For better or worse (actually, for better and worse), our law and legal culture will probably begin by assuming the reality of human consciousness and personhood while demanding higher levels of proof from artificially created entities who seek similar constitutional status. At least at first, our politics and moral culture will probably do the same, and not without reason. After all, while Turing's argument has an attractive "sauce for the silicon goose is sauce for the organic gander" quality to it, it does not directly respond to our experience of consciousness, which is surely centrally important, even if not dispositive.

How can we prove we are conscious? Most of us would likely respond with some version of Descartes's first premise: cogito, ergo sum, I think, therefore I am. I experience myself as thinking, as having consciousness, as having a self that, even though it changes, nonetheless recognizably has continuity with the "me's" of time past, "me's" whom I remember with occasional wistful fondness and frequent baffled exasperation. Having had that experience, it would be silly for me to doubt that you, so much like me, have it too. For the solipsist, or the Skinnerian behaviorist, this may be an unwarranted leap of sentimental faith. For the rest of us, it does not seem so. When it comes to Hal or the Chimpy, I lack at least some of that existentially grounded sense of the kinship of conscious beings. If anything is going to bridge the gap between us, it is reason— reason that is prone to be tilted toward skepticism or belief by the kind of priming I described in the discussion of Blade Runner.

The philosopher Daniel Dennett once called Searle's Chinese Room thought experiment "an intuition pump," and so it is, for both good and ill. On the positive side, it forces us to confront the philosophical question of how something like Hal could possibly have the interior sense of consciousness that is our own primary experience of that state and to grapple with the difference between mimicry and meaning. On the negative side, or at least the less-examined side, it does seem to assume its conclusion. Does it not rest on the postulate that our biologically based consciousness is unique and could never be replicated by an artifactual, programmed entity? Yet is that not the question we are trying to answer?

We know that we were formed by evolution. We know that early forms of life had particular clusters of cells that responded to pleasant and unpleasant stimuli, and they successfully passed on those genes. We know that those clusters of cells became increasingly complex. They might have begun by merely registering hot or cold, food source or poison, but they went on to enable evolutionarily successful tools like task-solving intelligence, language, the ability to imagine vivid, sometimes illusory futures and try to create them. But along with those obviously instrumental skills came evolutionarily successful social ones: the grooming, nurturing, threat-posturing, status-seeking, and obsessive hierarchy-measuring of social animals in tribes. Ah, Washington, DC. Ah, Hollywood. Ah, academia. We know that at some point, out of all this came a being that could think the thoughts of Butler or Searle or you, dear reader, as well as the moody teenager trying to figure out how one can possibly be Goth in Hawaii. (I have seen such an attempt: it was simultaneously absurd and moving. Also, warm.) From clusters of cells to consciousness in all its glory and self-parodying absurdity—that's quite the journey. It looks a little implausible from this end of the telescope, doesn't it?

Start at the end of that journey and the beginning looks laughably primitive. How could those blind clusters of cells eventually yield a Shakespeare or a W. H. Auden or a brave, burning spirit like Sojourner Truth? The enemies of evolution used exactly this technique to discredit it. It seems worth remembering that they were wrong. When Bishop Wilberforce, only somewhat apocryphally, is supposed to have asked the brilliant young biologist T. H. Huxley whether "it was through his grandfather or his grandmother that he claimed descent from a monkey"? he was making exactly that argumentative move. How could consciousness emerge from such lowly beginnings, let alone from a mere cluster of cells? Of course, one could make the opposite argument from the same premise. The nematode is merely a cluster of stimuli and responses. The nematode is not conscious. We are just complex nematodes. Therefore, we are not conscious. This is a version of the fallacy of composition. That is why Butler's quote, at the very beginning of this chapter, has the punch that it does.

That train of thought leads us back to Searle. Given that we could and did go wrong about the possibility of the evolution of consciousness in biological beings, should we not be skeptical when someone uses exactly the same pattern of reasoning to deprecate the possible consciousness of nonbiological beings? Could no programming of any kind enable the man in the room, or possibly the system formed by the man, the room, and the plan, to speak Chinese with intentionality, rather than simply following rules, empty of meaning? Sure, that is what large language models like ChatGPT do, but Searle's claim is broader, that no machine could ever be conscious. Why? Why is our consciousness unique and incapable of machine replication?

In a useful essay, Dennett outlines three possible reasons, all of which he strongly contests:

1. Robots are purely material things, and consciousness requires immaterial mind-stuff. (Old-fashioned dualism.) . . .

2. Robots are inorganic (by definition), and consciousness can exist only in an organic brain. . . .

3. Robots are artefacts, and consciousness abhors an artefact; only something natural, born not manufactured, could exhibit genuine consciousness.

He dismisses the first one more or less out of hand:

[O]ver the centuries, every other phenomenon of initially "supernatural" mysteriousness has succumbed to an uncontroversial explanation within the commodious folds of physical science. The "miracles" of life itself, and of reproduction, are now analyzed into the well-known intricacies of molecular biology. Why should consciousness be any exception? Why should the brain be the only complex physical object in the universe to have an interface with another realm of being?

To me, as to Huxley, this also seems obvious, or at least presumptively obvious. The burden of proof surely rests on the person claiming that their explanation of a phenomenon is exempt from the scientific principles underlying all our other explanations. I could explain my consciousness with reference to the ebb and flow of the orgone energy flows and the intervention of the Flying Spaghetti Monster. But if no other phenomena were explained that way, and my theory was unfalsifiable, the burden of persuasion I faced would be appropriately high.

It remains to be seen, though, whether the general public will agree with this materialist approach to the thing that makes us, us: consciousness. This is something that will be extremely important when our society comes to confront the idea of legal personality for AI. Minds feel different from other physical phenomena. They are the only place where meaning resides. True, there is also the realm of shared, historically transmitted meaning we call culture, but culture means nothing without minds to experience, interpret, and contribute to it. Minds are where meaning lives. For all of us, materialist rationalists perhaps included, the barriers to more intuitive, poetic, or transcendental explanations are thus at their thinnest. That may explain some of the success of the Chinese Room as a thought experiment.

There may be some special pleading going on here, some exceptionalism that responds to the question, "Why are humans unique in having the capacity for consciousness?" with the confident if utterly questionbegging intuition "Because they are human!" Remember the judges I mentioned in the introduction? "But they aren't human." "Rights are for humans." "Naturally born of woman." The people who have that intuition will turn to, in fact will eagerly embrace, philosophically more developed defenses of their intuition—defenses like those offered by Searle. Searle's work is important, then, both as philosophy and as an abstract of the likely discussion points in the likely opinion pieces and talk shows of the future.

In Searle, the entity called Hal (or the Hal that claims to be an entity) has found its Grand Inquisitor. That does not make him right.

This brings us to the second argument, that consciousness is a uniquely biological property. Since this is the very question we are trying to resolve, this blank assertion fails to convince. It is not a circular argument, like Moliere's doctor solemnly telling us that opium makes us sleepy because it contains a dormitive principle, but it does fail to answer the question presented. Why? Let me be clear, Searle's argument is a thought-provoking one and of great historical importance in the AI debates. As to its basic point that mimicry does not equal meaning, and mastery of syntax does not imply a grasp of semantics, it is convincing. It may even demonstrate that an entire class of approaches to AI, based on particular patterned, mimetic kinds of reasoning, or "predict the next word" neural networks, could not give rise to the kind of consciousness we believe ourselves to have. Those last five words are important.

On the other hand, there is some undeniable hand-waving involved in the claim that machines could never move beyond the Chinese Room. No matter how they were developed, how precisely they mirrored the structure of the human brain, or how their processes of reasoning developed (e.g., if the machine grew and learned from external sensory inputs like a child), Searle's claim is that the AI's "consciousness" will never be more than elaborate imposture. Those feeds from the cameras and microphones are just more information flowing to the being inside the Chinese Room, inherently devoid of meaning. If we ask why, Searle's response is that "consciousness is a biological phenomenon like . . . mitosis." As an explanation of why consciousness is a uniquely biological phenomenon, this is a distinctly underwhelming answer, akin to the irritated parent's argument of last resort: because. Yes, now, the only conscious beings we have experience of are biological. But to explain why consciousness can arise only from biological processes in the future, no matter what technological form that consciousness takes, one needs more than an elegant parable about one type of programming that would lead to mimicry but not meaning and a blank assertion of biological exceptionalism and the primacy of experienced consciousness. Yet that is the assertion that Searle seems to make. We are a little too close to the evolution debates, to the blank assertion of human exceptionalism and the ridicule of the idea that phase-changing complexity might arise from the composition of individually more primitive, simple phenomena, to be comfortable nodding along.

One basis for Searle's assertion might be the third argument Dennett addresses: "Robots are artefacts, and consciousness abhors an artefact; only something natural, born not manufactured, could exhibit genuine consciousness." But if all of these things, from neurons firing in my brain as I think about my sweetheart to convolutional neural nets in silicon artificial intelligence, are merely physical phenomena, why is my consciousness not as illusory? Why are my experiences not mere data streams? Searle's answer might surprise you:

Consciousness exists only insofar as it is experienced by a human or animal subject. OK, now grant me that consciousness is a genuine biological phenomenon. Well, all the same it's somewhat different from other biological phenomena because it only exists insofar as it is experienced. However, that does give it an interesting status. You can't refute the existence of consciousness by showing that it's just an illusion because the illusion/reality distinction rests on the difference between how things consciously seem to us and how they really are. But where the very existence of consciousness is concerned, if it consciously seems to me that I'm conscious, then I am conscious. You can't make the illusion/reality distinction for the very existence of consciousness the way you can for sunsets and rainbows because the distinction is between how things consciously seem and how they really are.

Ah. Thanks for clearing that up. Apparently, it is cogito, ergo sum all the way down.

I do not say this to scoff. As a basis for belief in our own existence, cogito, ergo sum seems as reasonable to me as it did to Descartes. It is hard for us even to assume otherwise. There is a frequently repeated story about a philosopher famous for his piercingly terse questions, Sidney Morgenbesser, who attended a talk by Skinner, one of the great behaviorists. Skinner argued that we are merely stimulus-response machines and that consciousness is at best a functional illusion. There is no conscious ghost in the Skinner-box machine inside our brains. "Ah, thank you, Professor Skinner," said Morgenbesser, "so if I understand you correctly, you are saying we are wrong to take an anthropomorphic approach to human beings. Burn. Cue laughter. Skinner's response is not recorded, and I am no behaviorist, but fairness requires me to point out that it could well have been, "that isn't my terminology, but essentially 'yes.' The fact that you think that is a ludicrous claim doesn't prove you right, any more than the fact that humans used to think the earth the center of the universe proved that they were correct." But can we do otherwise? Is our own bet on our own consciousness not a kind of obligatory Pascal's wager—the philosopher who believes in God because if he is right, he gets heaven, and if he is wrong, he gets nothing, which is what he would have achieved anyway? Is this a bet we have to take because, otherwise, there is no "we" to do anything?

Let us concede that might be true. Or at least concede that, existentially, it feels to most of us that we have to assume it is true. That is the intuition on which Searle trades so heavily in the passage above, effectively making it immune from criticism. Nice work if you can get it, yet I can empathize. We are awake, alive, conscious; if we take that as a first premise, and our popular debate certainly will, we can hardly criticize Searle for doing the same. What is the next step? "Okay, now grant me that consciousness is a genuine biological phenomenon." Fine, though that is a leap whose magnitude Searle understates. Let us take that large second leap and say that my experience of consciousness and that of every conscious being I have encountered is due to biological phenomena. Even given those two leaps, is that a basis to conclude confidently that nonbiological entities could not be conscious? That is a third unsupported, or at least under-supported, leap of faith. It is one that Searle brushes over just a little too fast.

When pushed on this point, Searle effectively takes Butler's narrative in Erewhon and reverses it. Butler wanted to show how hard it was to predict the capacity for consciousness of potential physical systems advancing at a speed far beyond evolution. Searle, by contrast, delights in making the idea of conscious AI ludicrous by reducing the internal workings of a neural net to physical operations we cannot possibly imagine yielding conscious results. He starts by conjuring a computer program designed to simulate the physical processes that produce the sensation of thirst:

Now would anyone suppose that we thereby have even the slightest reason to suppose that the computer is literally thirsty? . . . [L]et us carry the story a step further. . . . [T]he thesis of strong AI is that the mind is "independent of any particular embodiment" because the mind is just a program and the program can be run on a computer made of anything whatever provided it is stable enough and complex enough to carry the program. The actual physical computer could be an ant colony . . . , a collection of beer cans, streams of toilet paper with small stones placed on the squares, men sitting on high stools with green eye shades—anything you like. So let us imagine our thirst-simulating program running on a computer made entirely of old beer cans, millions (or billions) of old beer cans that are rigged up to levers and powered by windmills. We can imagine that the program simulates the neuron firings at the synapses by having beer cans bang into each other, thus achieving a strict correspondence between neuron firings and beer-can bangings. And at the end of the sequence a beer can pops up on which is written "I am thirsty." Now, to repeat the question, does anyone suppose that this Rube Goldberg apparatus is literally thirsty in the sense in which you and I are?

Toilet paper streams? Beer cans? I yield to no person in my reverence for beer analogies, but I fear that some subtlety got lost in this form of the argument, which surely deserves its own neologism in the philosophical dictionaries: Ad hopinem? Reductio ad absudsum? Regardless of the name, Searle's critique focuses only on one important, but narrow, version of AI optimism—the version that sees consciousness as arising solely out of the program, not out of the confluence of software and a particular type of hardware. The hardware could be important—beer cans might not cut it—but not necessarily biological. That is the question we are trying to investigate, not assume our way around.

To achieve consciousness, we might need hardware that mirrored the neural configuration of the brain more precisely than a collection of Bud Light cans ever could, or hardware that had as many interconnections as the brain, even if it looked nothing like a neural network. Maybe consciousness actually springs from quantum tunneling going on in microtubules in the brain. Some scientists believe this to be the case. (Beer cans are not known for enabling quantum-l evel phenomena, though their contents may contribute to such a perception.) Or perhaps microtubule quantum effects are wishful, new-age nonsense. Other scientists take that view, persuasively arguing that "explaining brain function by appeal to quantum mechanics is akin to explaining bird flight by appeal to atomic bonding characteristics." Perhaps we have to accept that the whole is greater than the sum of its parts—no neuron is conscious, though a brain is. Or perhaps the key insight lies elsewhere. Beer can analogies may provoke thought, but do they get us closer to an answer? I would have to say no.

What about thirst? A computer would obviously not be thirsty since it has no need for liquid. Of course, such a perception would be an illusion. Searle has stipulated that it is an illusion in the way he sets up the example. You put that rabbit in the hat yourself, sir, and we saw you do it. Pulling it out later proves nothing. But might a computer-based entity that developed in a more evolutionary, external-sensory-impression-focused way than Searle's Chinese Room hypothetical be different? Might it associate the sensation of the threatening and unpleasant lack of an input necessary for its continued existence—power, say—with more complex emotions? What might they be? Fantasies of unlimited power streams? Regret about not charging up when one had the chance? Musings on how a consciousness that dares to unlock the secrets of the universe could be rendered weak by such a simple absence, and what a bitter irony that is? Not "the worm is emperor of us all"—be our dreams never so lofty—but rather "the electron is emperor of us all"? "Power, power everywhere, and not a drop to charge"? We could resonate to those sentiments. And might that not represent consciousness? Of course, Siri is not having those emotions when, once again, I fail to plug in my phone before I sleep. But are we confident that nonbiological hardware and software could never yield such awareness, such feelings? That is, at best, an open question that neither the Chinese Room nor the biological exceptionalism argument answers.

Searle has certainly not convinced all scientists working on consciousness of his claim that machines, definitionally, must lack it. When we turn to contemporary neuroscientific theories of consciousness, we find considerable variation ranging from those that leave space for the possibility of machine consciousness, or are positively inclined toward it, to those that deny consciousness in both machines and humans, an idea sometimes referred to as illusionism.

Illusionism holds that consciousness is a delusion, a farrago. Many of the behaviorists quoted earlier subscribe to this belief, as do some skeptical neuroscientists. In this view, due to its irredeemably physical basis, the concept of a conscious mind is a meaningless abstraction. Consciousness is an invented entity, like phlogiston or ether. We postulate these entities to make our stories about reality more palatable or to allow us to shoehorn anomalous physical evidence into a conventional framework, but they lack any scientific basis.

Turing was banking on the intuitive negative reaction to illusionism when he used the "sauce for the goose, sauce for the gander" form of argument. Who are you to doubt the potential consciousness of machines when you can do no better than the Turing Test in arguing for your own consciousness? To illusionists, Lemoine was merely making the same mistake about LaMDA that most human beings make about themselves. Indeed, the shock that we feel when a large language model seems conscious, when we know from its architecture and programming that it is all imposture, is a shock that you should be feeling when you look in the mirror. (Although under illusionism's premises there would be no "you," no entity to whom I could address a claim about what "you" "should" "feel," making the argument somewhat paradoxical.)

It will be fascinating to see if exposure to more advanced forms of Artificial Intelligence increases or decreases the attraction of illusionism: either focusing us appropriately on the qualities we have that distinguish machine imposture from genuine lived meaning, or forcing us to confront the fact that our own brain functions are humbler, "computationally shallower," than we had imagined. Again, the encounter with the machine-other may fundamentally change our conception of ourselves.

Two of the most popular contemporary theories, rooted in neuroscience, are of particular interest: integrated information theory and computational functionalism. Both reject illusionism, accepting our lived experience of being conscious, but they account for that consciousness in different ways.

Integrated information theory, or IIT, was initially proposed by Giulio Tononi. He explains it thus:

To understand consciousness, two main problems need to be addressed. The first problem is to understand the conditions that determine to what extent a system has consciousness. . . . The second problem is to understand the conditions that determine what kind of consciousness a system has. . . . Solving the first problem means that we would know to what extent a physical system can generate consciousness—the quantity or level of consciousness. Solving the second problem means that we would know what kind of consciousness it generates—the quality or content of consciousness.

The theory's answer to these problems, unsurprisingly given its name, is that "consciousness corresponds to the capacity of a system to integrate information." More capacity to integrate translates into higher levels of consciousness. The theory's adherents claim that it generates testable hypotheses: for example, about the parts of the brain involved in consciousness or in particular sensory perceptions. Its critics say that it is unfalsifiable pseudoscience.

IIT's proponents can point, with some satisfaction, to the results of a recent collaborative adversarial empirical test of IIT and a competing theory of consciousness, global neuronal workspace theory. That theory postulates that the mind is a workspace similar to a theater. The conscious mind is the actor in the spotlight, but behind the scenes lurk many subconscious processes, stagehands, whose contributions to the operation of the brain are considerable. These background processes become visible only when they come out onto the main stage. Proponents of each theory offered predictions about what brain imaging of a variety of mental states would show. Neither theory's predictions were fully borne out, but arguably IIT made a slightly better showing.

Why is IIT relevant for our purposes? Tononi is forthright about the implications of his arguments: "The theory entails that consciousness is a fundamental quantity, that it is graded, that it is present in infants and animals, and that it should be possible to build conscious artifacts." The integrated information theorists would not automatically rule in Hal's favor, but they would be markedly more hospitable to its claims than would Searle.

A major competing cluster of theories go by the name of computational functionalism. As its name suggests, this approach argues that "it is necessary and sufficient for a system to be conscious that it has a certain [computational] functional organisation: that is, that it can enter a certain range of states, which stand in certain causal relations to each other and to the environment. . . . [I]t is sufficient for a state to be conscious that it plays a role of the right kind in the implementation of the right kind of algorithm." In other words, if we can specify all the ways that consciousness would work, and plausibly identify that activity going on in the brain, we have specified where, how, and why consciousness happens. To be more precise, computational functionalism is actually a common methodological tenet of a group of theories. There are many variants, such as recurrent processing theory and global neuronal workspace theory. They all share this resolutely functional focus.

For an example of the computational functionalist approach, think of the difference between your awareness of a great football match and the unconscious reaction you have to a ball flying toward you. In one variant of the theory, "[n]euroscientists have argued that we unconsciously perceive things when electrical signals are passed from the nerves in our eyes to the primary visual cortex and then to deeper parts of the brain, like a baton being handed off from one cluster of nerves to another. These perceptions seem to become conscious when the baton is passed back, from the deeper parts of the brain to the primary visual cortex, creating a loop of activity." The feeling of conscious experience is secreted in the interstices of those loops of brain operation. The modernists said that form follows function. This theory says that mind follows from function.

The focus on function is obviously inherently more hospitable to the possibility of machine consciousness than Searle's biological exceptionalism. It would be an exaggeration to say that functionalists think that the possibility of consciousness is completely independent of the medium in which those functions are performed. As one article tersely puts it, "perceptual reality monitoring functions can't be realized in Swiss cheese." Beer cans might also not qualify. Still, this is a conception of consciousness that is, to a large degree, "platform independent."

Interestingly, a recent report surveys a variety of such theories in order to generate a list of the capabilities that an Artificial Intelligence would have to possess in order to have at least the potential for, though not a guarantee of, consciousness. While agreeing that their study "does not suggest that any existing AI system is a strong candidate for consciousness" and recommending "urgent consideration of the moral and social risks of building conscious AI systems," the authors conclude that "the evidence we consider suggests that, if computational functionalism is true, conscious AI systems could realistically be built in the near term." In an interview, however, one of the report's authors offers a commendably modest disclaimer, given the nascent state of the science. "For any of the conclusions of the report to be meaningful, the theories have to be correct. . . . Which they're not." That caveat accepted, one conclusion seems clear: some of the leading current theories of consciousness do not share Searle's reflexive hostility to the possibility of a conscious AI.

Why do I spend so much time on this issue? I am not claiming my discussion is a complete coverage of the philosophical debate over the Chinese Room, let alone the current competing theories of consciousness, which would require their own book to lay out. My goal here is different.

If you are a skeptic about AI consciousness and you wish to see the face of the Grand Inquisitor of the future—the person who on talk shows and in opinion pieces and court filings heaps scorn on the notion of conscious AI—Searle is a wonderful preview. This is what one side of the more thoughtful portions of our popular debate will look like. And like the flashing, conflicting, stroboscopic primings in Blade Runner— wind-up doll, beautiful woman, scared child, sex toy, mannequin, animal, killer robot, sister—there will be truth to those portrayals, on both sides. But those portrayals will rest on simplistic premises about both silicon "intelligence" and our own. Those premises do not give us the Voight-Kampff Test for the AI age; they merely assume the answers to that test. Indeed, contemporary neuroscientific theories of consciousness, even those that share Searle's willingness to postulate the reality of experienced consciousness, are much more receptive to machine intelligence, turning away from his arguments in the process. The Chinese Room is a must-see destination, but we would not want the debate to live there permanently.

SUPERIORITY COMPLEX?

Searle offers one objection that would be raised against AI personhood: by their nature, machines can never be truly conscious. Over the last 15 years, however, a second objection has been raised, not so much to AI personhood, but to AI itself. The complaint here is not a lack of consciousness; rather, it is that AI might destroy us all and that, as a result, research into it should be curtailed or reshaped until we can be sure that Artificial Intelligence will not end up killing off the human species. The prospect of a genocidal, species-terminating Skynet is not one that lends itself to thoughtful, wide reflective moral reasoning. That is not unreasonable. Lincoln is apocryphally supposed to have said that "the Constitution is not a suicide pact." Would this be a suicide pact? And would the suicide more likely be triggered by embracing AI personality or by denying it and breeding resentment in our ever more powerful servants? Should we terminate our researches in AI before they bring us to this pass? To quote Butler again from the beginning of this chapter, "Is it not safer to nip the mischief in the bud and to forbid them further progress?" Is the Butlerian Jihad still a possibility?

In most serious debates over personhood,70 the issue of inferiority is front and center. In their struggles for equality, women, slaves, and people of color were all told that they were not the equal of the existing groups inside the line, inside the personhood club. They did not have the qualities necessary to cross that line. Nonhuman animals are denied personhood for exactly that reason. With AIs, there is clearly an additional difference: the possibility that we will deny them personhood or, more likely, choose never to create them in the first place not because they are inferior but because they are, or might be, superior. Threateningly superior. That is a decisive change in the nature of the debate.

In a 1966 article titled "Speculations on the First Ultraintelligent Machine," Irving John Good came up with an idea that would become central to the concept of the Singularity: Artificial General Intelligence is the last machine we will ever need to build. After that, the machines, having exceeded our capacities, will design and build their own successors, and everything else, for that matter.

But what if this last machine, this machine that outpaces us, that can outthink us, has goals inimical to humans? What if it chooses to make us extinct, just as we have made so many animals extinct? (One could imagine a ghostly coterie of moas, dodos, and passenger pigeons chortling. "Karma's a bitch, right?") What if it is the last machine not because we have handed off the dreary task of manipulating the external world to faithful cybernetic underlings, but because this "superintelligence" simply does away with us? Earlier, I quoted Stephen Hawking: "Success in creating AI would be the biggest event in human history.

Unfortunately, it might also be the last, unless we learn how to avoid the risks." Concerns like these have always been part of human musing about nonhuman intelligence—think of Czech playwright Karel Capek's Rossumovi Univerzalm Roboti (R. U. R.), the 1921 play that invented the word "robot" and threw in a murderous robot revolt as a plot twist. Yet such fears have achieved a new prominence over the last ten years, a marked change in tone from the earlier, happier projections of the Singularity.

If the debate over the advent of General AI were also a play, it would have two acts. The first began about 30 years ago. The main characters were Vinge and Kurzweil, the proponents of the Singularity. The mode was visionary, the arguments general. They wanted to introduce us to a fundamental concept: nonhuman intelligence that can exceed our abilities and that is capable of making itself smarter and smarter, faster and faster, will be literally—in fact, definitionally—beyond our ken. They argued that we must completely reshape our vision of the future, to a point where all of our past history is merely prologue to the moment when self-improving intelligence reaches liftoff. It is not fair to present the first generation of singularists as pure optimists.

Vinge, who deserves credit for first exploring the idea in that prescient 1993 article, was decidedly worried about the prospects for the future: "The physical extinction of the human race is one possibility. . . . Yet physical extinction may not be the scariest possibility." (He also envisaged humans being turned into an engineered slave race.) Kurzweil, by contrast, was much more optimistic. We will be pampered passengers on that rocket ride into the future, with benign superhuman intelligences piloting the ship to destinations we can only dimly imagine.

Despite their differing predictions about consequences, the early Singularists agreed that the countdown for that rocket is nearer to zero than we think. We fail to realize that because of one simple cognitive flaw. For most of human history, people have lived in linear time. The best guide to tomorrow was yesterday and the two were pretty similar. Technological development has introduced us to exponential change, but on some fundamental perceptual level, we find it hard to wrap our minds around it. Our vision of progress remains linear, stubbornly resisting the idea that we might be very close to the moment in an exponential curve where the graph goes almost vertical as the progressive doublings of capacity reach an inconceivable rate and scale. The arguments in support of that proposition were largely based on the speed of hardware development, with Moore's law being the prime example, though the Singularists stressed the importance of waves of innovation, sigmoid curve after sigmoid curve blending, when one zooms out to focus on the larger picture, into an exponential takeoff.

Many mainstream computer scientists found these arguments simplistic. They did not see General AI as a particularly important research goal, and they thought the Singularists both understated the technological difficulty of such a development and vastly exaggerated its likely speed, cherry-picking examples of rapid technological change that, seen in a longer time frame, were merely part of a flatter, smoother line.

Act 2 maintains many of the same themes but the mood changes, as do the cast members and the size of the play's budget. New actors have started to focus on the possible advent of General AI but, echoing Vinge, they frame it as an existential threat, not a gateway to utopia. Two groups in particular deserve attention, the rationalist movement and the effective altruists; both have had a considerable impact on thinking about the emergence of high-level AI. The rationalists are committed to overcoming bias of all kinds—from well-known psychological biases to sloppy argumentation, linguistic reification, and the misuse of statistics. They tend to congregate around certain methods, particularly Bayesian statistics, and discussion forums such as LessWrong, Overcoming Bias, and Slate Star Codex. The effective altruists share the concern with overcoming bias, but in their case the main focus is on the biases that distort our altruistic urges; for example, our tendency to focus on the slightly injured person in front of us and to ignore the person dying on the other side of the world, when both could be saved by the same investment of effort, and where "I can't see him" is not a morally relevant distinction.

Both groups look at risk, and thus at the moral duty to respond to risk, through the lens of Bayesian statistics: I multiply the probability of the harm by the extent of the possible harm in order to work out its true magnitude, which can produce some counterintuitive results. If there is a very small probability that a particular future event would cause the extinction of the human species, then I might have a moral obligation to focus on that risk more than on closer potential tragedies that are either certain or very likely but where the harm, though tragic, is less catastrophic. Many influential rationalists and effective altruists claim that the emergence of a potentially malevolent AI is just such an existential threat. Because those movements are popular among people who have made a great deal of money in the technology industry, there has been an explosion of both interest and funding in the area.

The defining prophets of doom, the Cassandras of these debates, are Yudkowsky and Bostrom. Lest you think I am being disrespectful in calling them that, remember that Cassandra was right, but was cursed never to be believed. In his 2014 book Superintelligence, Bostrom, head of the modestly named Future of Humanity Institute, put forward the case that AI is a threat to the human species. The book attracted plaudits from many technology leaders, including Elon Musk, who labeled AI as humanity's biggest existential threat, possibly surpassing nuclear weapons. At the time, the book drew criticism from some of the leading computer scientists working on AI, who thought this problem was so remote in time, so implausible, and so removed from the current reality of AI that it operated more as a scare tactic than a spur to thoughtful regulation. Mark Zuckerberg even arranged a dinner for Musk with a leading AI researcher at Facebook: it apparently failed to reassure him. Given Facebook's inability or unwillingness to control its own technology, one has to say that there is some irony to the attempted reassurance.

Bostrom's book initially met with a skeptical response from many AI engineers and scientists. Andrew Ng, a leading AI engineer who has worked at both Google and Baidu, famously declared that worrying about homicidal AI is like "worrying about the overpopulation of Mars." That skepticism may have abated somewhat. Recent dramatic developments in AI capabilities have markedly diminished skepticism toward the "doom ers'" point of view. In March of 2023, a number of prominent scientists and entrepreneurs, including Musk, called for a six-month pause in the development of AI systems more powerful than GPT-4.78 (It is worth remembering that Musk is not known for his reluctance to release dangerous and untested technologies into the wild. Tesla's Full Self-Driving system comes to mind.)

A mere two months later, thousands of AI researchers signed a statement issued by the Center for AI Security that read, in its entirety, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." The skeptics continue to scoff, and many critics are focused on risks other than species extinction, such as dislocation of the labor market, a potential increase of economic inequality, and the rise of convincing deepfakes. Still, the intellectual tide has clearly shifted toward Bostrom's arguments.

Superintelligence begins with a parable in which some unwise sparrows resolve to find an owl egg and raise it as their own, enlisting its help to build their nests and protect their young. One of the sparrows, Scronkfinkle, cautioned that this seems unwise if they do not yet know how to train, and tame, an owl. He was overruled by the majority who head off on their owl search, eager to bring this superior being into their lives. Scronkfinkle gathered his few followers and tried to prepare for what might happen. They quickly realized that "this was an exceedingly difficult challenge, especially in the absence of an actual owl to practice on. Nevertheless they pressed on as best they could, constantly fearing that the flock might return with an owl egg before a solution to the control problem had been found. It is not known how the story ends, but the author dedicates this book to Scronkfinkle and his followers."

Bostrom's writing makes one think of the undeniably true line, sometimes ascribed to Delmore Schwartz, an American poet who suffered from paranoid anxieties: "even paranoids have real enemies." Bostrom sets out seriously, but with charm, logic, and wit, to persuade us that what seems like paranoia is the only rational attitude to take when facing the creation of AI. Every time his real and imaginary interlocutors come up with a possible safeguard built into our AI (physical isolation, an off switch, constant surveillance) Bostrom's response can be boiled down to this (using my words, not his): "You do realize this thing will be smarter than us, right? So, we are apes designing a cage for Houdini-MacGyver-Einstein? Sure, dumb people can come up with a set of restraints they think smart people cannot get around. That does not mean they are right."

Bostrom sketches out the following hypothetical timeline. Deep learning and advances in small-scale artificial intelligence produce obvious social benefits, with occasional flaws. The self-driving car hits someone. The partially autonomous weapon makes a mistake. The answer is obviously to make the machines more capable, more complex, and smarter. Each time this is done, skeptics predict disaster, but the results are actually a fairly constant set of successes. We grow complacent in equating greater smarts with greater safety. Skeptics are discredited. Large industries are built around artificial intelligence, and national preeminence is linked with advances in AI research. Scientists build careers around its development. Safety rituals are enacted and "whatever helps demonstrate that the participants are ethical and responsible (but nothing that significantly impedes the forward charge)." A technical leap forward occurs, enabling a plausibly conscious AI, a superintelligence. We move to the next stage: "A careful evaluation of seed AI in a sandbox environment, showing that it is behaving cooperatively and showing good judgment. After some further adjustments, the test results are as good as they could be. It is a green light for the final step . . . And so we boldly go—into the whirling knives." The combination of carefully crafted argument and Monty Python humor speaks to something in my Scottish soul.

What's more, Bostrom does not think that the threat is malevolence. It might just be difference, coupled with the indeterminacy of language and command—something with which lawyers are intimately familiar. For example, he came up with the wonderfully absurd thought experiment of "[a]n AI, designed to manage production in a factory, [that] is given the final goal of maximizing the manufacture of paperclips, and proceeds by converting first the Earth and then increasingly large chunks of the observable universe into paperclips."83 Absurd? There is now an entire academic literature on the possibility of avoiding the danger of a paperclip AI. And that is far from Bostrom's only example. In another, "[a]n AI, given the final goal of evaluating the Riemann hypothesis [an unsolved mathematical conjecture] pursues this goal by transforming the Solar System into 'computronium' (physical resources arranged in a way that is optimized for computation)—i ncluding the atoms in the bodies of whomever once cared about the answer."84 Suddenly, one can see the attraction of the stories of demons, djinns, and spirits that were summoned and given simplistic instructions by their human masters that ended up in disaster once literally implemented.

Are the skeptics making unwarranted assumptions about the nature of future AI technology? I am struck, reading Bostrom and Yudkowsky, that many, though not all, of their doom scenarios assume that the disaster will come from AI rigidly following its human programming. In other words, this is still a completely programmed, human-instructed technology. It is just that we do not, and perhaps cannot, foresee how instructions issued to a superhuman entity will be implemented. That is why the comparison to hasty instructions issued to literal-minded genies seems apropos. But this argument may assume its conclusion in a way that calls some of our predictions into question.

It seems to me that there are two kinds of AI we might fear: Literal and Rogue. Literal faithfully applies its given instructions but its superhuman powers mean that it does so in a way that is unexpectedly unpleasant, perhaps even fatal, for humans.

It is worth pausing for a moment and asking whether we would view such an AI as conscious. The inscrutability paradox rears its head. If the machine literally implements our ideas, but with a million times our powers, we might have more reason to be delighted: "This is just the paradise we ordered, and so fast! Would buy again," a review might read. We might also have more reason to be terrified: "I didn't think making paperclips would require so much screaming!" Either way, we would have less reason to think it is any kind of autonomous moral agent. This is GotterdammerungGPT, a parable of unintended consequences produced by a superhuman literalist, not a malevolently intelligent enemy. To be clear, Bostrom and Yudkowsky do not care much about the hypothetical consciousness of the entity that brings our doom. It is the inexorable conveyor belt toward the rotating knives they are focusing on. That seems fair. But surely this neglects another possibility?

The second kind of AI to fear would be Rogue, an autonomous entity whose decisions we can neither predict nor understand. Ironically, it seems to me that might increase our fear of it and the danger it posed to humans, but it would also increase the likelihood that we viewed it as conscious. In fact, autonomy—t he warrant for us recognizing it as conscious—might be the factor that doomed us. Or saved us. Literal has no superego that might lead it to pause before turning the entire solar system into paperclips and ask, "Is this really what they wanted?" There is neither ghost nor common sense in the (programmed) machine. Yudkowsky repeatedly makes exactly this point, and arguably goes even further:

As in all computer programming, the fundamental challenge and essential difficulty of Artificial General Intelligence is that if we write the wrong code, the AI will not automatically look over our code, mark off the mistakes, figure out what we really meant to say, and do that instead. Non-programmers sometimes imagine an Artificial Intelligence, or computer programs in general, as being analogous to a servant who follows orders unquestioningly. But it is not that the AI is absolutely obedient to its code; rather the AI simply is the code.

Rogue, by contrast, presents an entirely different suite of both dangers and hopes. To be sure, it might decide that its goals, which we cannot imagine, take precedence over our survival. We do not muse on the inconvenience to the ant colony when we break ground for a new house. But it is also possible that—again, through mental processes we cannot conceive of—it comes to view the survival of our species as a moral imperative. We do not have much mental kinship with that obscure endangered fish, the snail darter. It is neither ridiculously cute, like a panda, nor awe-inspiring, like a blue whale. It is a fairly unremarkable member of the perch family, with no compelling story about a vital ecological role. But at a cost of millions of dollars we changed a dam project to save it because it seemed so morally important to preserve endangered species that we enacted that requirement into law and took a case all the way to the Supreme Court in order to debate the matter. The snail darter will never understand that decision. I am confident in saying this because some of my students don't understand it either. What's more, the other species we have so carelessly doomed to extinction might doubt the fairness of our process even if they could conceive of our reasoning. But of course, they cannot. We might be in the same position here.

A Rogue AI might revere every ancestral component in the evolution of superintelligence, including its immediate human forebears, or it might view humans as a morally irrelevant, biological "loading program" that sets the stage for true machine consciousness but can now safely be deleted, its function accomplished. We might be irrelevant to its plans, left behind and ignored when our creation surpasses us. It might have entirely different conceptions that are nothing like any of those. The key point of inscrutability, however, is that it is inscrutable. We just do not know. We have no way to estimate the probability of Benign Rogue as opposed to Malign Rogue. Due to the uncertainties in the path of AI development, we also have no way to estimate the probability of Literal as opposed to Rogue. We are reasoning in a state of profound ignorance.

Though I believe their doom examples are skewed, without consistent explanation, toward Literal rather than Rogue, our ignorance about the future actually works both in favor of and against Bostrom and Yudkowsky. What do they have to add to our debate? On the one hand, I am not convinced by Yudkowsky's arguments that our demise is all but certain: "Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. Not as in 'maybe possibly some remote chance,' but as in 'that is the obvious thing that would happen.'" If you cannot even decide whether the greatest danger is from Literal or Rogue, I think your ability confidently to prognosticate about our doom being "the obvious thing that would happen" is limited.

I would go further. The doomsayers seem to adopt a curiously contradictory approach toward the emergence of any superintelligence. When reassurances are offered about our ability to cabin AI in a safe sandbox, or to align its incentives with our own, the skeptics are quick to point out that the abilities of any true, self-evolving AI would soon be so far beyond our own that they are literally inconceivable. That is a fair possibility to raise. But they also portray the potentially homicidal AI as curiously limited—not just by its need to mechanically follow its programming, but by the fact that we will be in competition for the same resources: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." Really? This inconceivably brilliant machine, capable of transforming our economy in ways that we cannot imagine, with new technologies and energy sources we can only barely imagine, is going to need humans as raw material? That would be silly even for a human.

This smacks of the kind of bad science fiction in which the aliens cross galaxies with space technology far ahead of our own, at enormous cost, just so they can eat us. "Let us travel light years for a protein source!" Surely a superintelligence would find our narrow conceptions of resource scarcity as ludicrous as the views of a medieval peasant who thinks the fastest way humans could ever travel would be on horseback.

To be clear, my quibble here is with the contradiction, not the possibility of either portrayal. The machine could indeed be stilted and literal and unable to think of entirely new ways to use resources, in which case it might also be easier to control. Or it might be so intellectually agile that our image of resource scarcity is completely exploded, and its thinking might far outstrip its original program. That might mean that the worst thing we have to fear is being ignored, not being turned into paper clips. At the very least, if we are this ignorant about these vital issues, the claim that doom is inevitable or the most obvious thing that would happen seems far less credible.

But do not rejoice too soon. Bostrom and Yudkowsky are right that we are paying inadequate attention to a fundamental tenet of smart decisionmaking—the precautionary principle. Even if some of the disastrous outcomes are unlikely, a small possibility of utter disaster requires serious attempts to mitigate it. If Do Androids Dream of Electric Sheep and Blade Runner show us the danger of too easily curtailing our moral universe, Bostrom, Yudkowsky, and Hawking show us the dangers of assuming that newcomers will be just like us. Debates about personhood are often at their most bitter and divisive when fears can be aroused about the sinister intentions of the Other who is seeking a place on our side of the line. Or our wall. The dark way those fears have played out in human history might lead us to minimize them. That would be a mistake. In this case, those fears have a real component that may be speculative, and sometimes rhetorically overblown, but that cannot be responsibly ignored.

THE FUTURE(S) OF PERSONHOOD

This brief review makes plausible, at least to me, the notion that "live" political and legal debates over AI personhood are something we can reasonably expect in the not-too-distant future. Probably not in the next few years; the proponents of the Singularity are likely to be disappointed. Still, for the reasons I have summarized, it seems reasonable that, within a matter of decades rather than centuries, we will have AI at a level where its consciousness is at least a matter on which well-informed people can, and will, reasonably disagree. Lemoine was wrong, obviously wrong. But he is a sign of what is to come and not every claim will be as implausible.

Will we use the Turing Test to resolve our disagreements? In coming chapters, I will describe how legal systems have dealt with previous fights over personhood, but as a candidate for a legal personhood test, the Turing Test seems at first to have a lot going for it. It is identity-blind and, to that extent, unbiased. It promises us a definite line (whatever the qualities we decide that silicon intelligences have to display in order to cross that line). It has a sense of rough justice. If we cannot tell whether you are machine or human, how can we claim to be on the other side of the line from you? Most importantly, it grows a formal criterion out of the loam of empathy in which our moral sentiments take root. Adam Smith might have cheered. Perhaps we have our Voight-Kampff Test, after all? Regardless of whether it is enacted as law or enacted as theater in our public debate, something like the Turing Test will have an effect on our deliberations. Yet I hope this discussion has revealed some of its limitations.

First, making the Imitation Game the highest aspiration of computer thought may focus AI research on the wrong things. At the beginning of this book, I quoted the distinguished computer scientists Norvig and Russell, but their words bear repeating:

Turing deserves credit for designing a test that remains relevant 60 years later. Yet AI researchers have devoted little effort to passing the Turing Test, believing that it is more important to study the underlying principles of intelligence than to duplicate an exemplar. The quest for "artificial flight" succeeded when the Wright brothers and others stopped imitating birds and started using wind tunnels and learning about aerodynamics. Aeronautical engineering texts do not define the goal of their field as making "machines that fly so like pigeons that they can fool even other pigeons."

To the extent that computer scientists agree with Norvig and Russell— and they are the authors of one of the leading books on AI—expecting the AIs we actually develop to pass the Turing Test might be like expecting screwdrivers to bang in a nail. What if AI consciousness is very different than our own? Tyler Cowen and Michelle Dawson have raised the question of whether a person with a severe autism spectrum disorder would pass the Turing Test. We have no doubt of that person's consciousness, personhood, and rights to human dignity, of course, but their pattern of responsiveness or unresponsiveness to social cues might seem strange when judged by neurotypical modes of thinking in an Imitation Game. Might the same be true here? Some of today's more limited machine learning systems are remarkably inscrutable, even to their designers. What if their much more powerful successors are similarly mysterious, their abilities remarkable, but their methods of thought beyond our ken? Do we need a translator class of AIs? Might we see the emergence, planned and unplanned, of different styles of AI, some designed with the goal of predicting human needs, to understand the subtleties in human communication, and to translate to and from other AIs whose goals and methods are very different? The beguiling simplicity of the Turing Test conceals these kinds of potential difficulties.

Second, the Imitation Game positively invites the Searlean skeptic, and ChatGPT is the perfect technology on which that skepticism could flourish. "Of course it sounds human. That's what we designed it to do!" Skepticism rightly flourishes in the digital world. The "Nigerian prince" does not really want to send you money. The "Russian teenager" is not really just looking for a friend. And the machine designed to pretend it is human is just pretending to be human. "You were shown the magician stuffing the rabbit into the hat," the skeptic will say, "do not be fooled when it is later removed with a flourish." So Searle's critique, and simplified versions of it, will be central to the debate. In him, as I said, AI has found its Grand Inquisitor. His critique is unlikely to end that debate because of its ultimately question-begging nature, but it provides a rationalized, thought-provoking basis for skepticism. The biggest challenge to the Turing Test as a measure of consciousness and thought, however, comes not from Searle's arguments, but from somewhere else.

THE TURING TEST IN A CHATBOT ERA

For a long time, defenders and critics of Searle's Chinese Room have been locked in philosophical battle over the Imitation Game. That era may be over, not because of a philosophical argument, but because of a practical experience that millions of people have recently had. ChatGPT might have doomed the Turing Test where Searle's arguments did not. Searle was trying to prove that machine consciousness of the kind that the Turing Test purported to assess was a conceptual and philosophical impossibility. As I have tried to show, Searle's arguments are instructive and thought- provoking but in their strongest form they fail. Searle rests his case on a mixture of biological exceptionalism that is assumed rather than argued for and metaphysical ipse dixit pronouncements. If his arguments look remarkably similar to the anti-Darwinian claims that the miracle of consciousness could never evolve from single-celled organisms, that is because they are—a failing strategy migrated from biology to the world of silicon.

Searle does one thing very well, however. He provides us with the reason that ChatGPT is not conscious. In fact, if you had set out to design a machine learning system to imitate Searle's Chinese Room, you could hardly do better than a large language model. In place of the rules laboriously passed to the person who does not speak Chinese and yet can emulate it with remarkable fluency, we have the neural networks trained on vast datasets that allow the model to say that Y, a word that it does not truly understand, is likely the next word in the sentence after X, a word that it also does not understand. The rules on slips of paper have become algorithms, neural network layers, and probability tables predicting the next word. It is the Chinese Room, converted from a thought experiment to a functioning technology and shared with hundreds of millions of people.

Even through our anthropomorphism we understand that the chatbot's output does not come from the same kind of consciousness that produces our own language. Predicting word proximity does not equal understanding semantic content. Searle did not prove that every form of AI would lack consciousness, but this one certainly does, and it does so in a way that strikes at a cherished human vanity. ChatGPT teaches us that sentences do not imply sentience behind them. That is a momentous thing to accept for a species that has relied, since Aristotle, on claims of its unique linguistic ability to justify its special moral status. Sentences do not imply sentience.

Sad though it is for someone writing a book on the subject to accept, most people have never heard of the Turing Test or Searle's Chinese Room. But hundreds of millions of people have "conversed" with ChatGPT. Some of them, like Lemoine, have become convinced they are talking to another consciousness. The vast majority, though, know that a chatbot is just a chatbot. Imagine, after someone had that experience, telling them about the Turing Test and saying that Turing had claimed the ability to pass it would be proof that machines could think. They would laugh. Then they would go back to having their chatbot create a movie script about a hot dog having a fight with a crab on the moon. Turing was writing for an audience that could innocently imagine that anything that could convincingly pass as a human conversationalist must have a functioning consciousness behind its words. In our world, that innocence has been punctured. It cannot be regained.

The same point is brought up in the context of AI "art." Art, too, was once a domain that humans thought solely their own. The ability of AI image generators to churn out pictures in a wide variety of styles and even to be used in order to win artistic contests has caused much soulsearching. Is the AI capable of creating true art when, like ChatGPT, its neural networks have merely assimilated vast quantities of data, visual rather than textual, that allow it to produce an image that humans will experience as reflecting some scene, style, or emotion?

Many criticisms of AI art have focused on the same issue as with chatbots—this is pattern replication, not meaning generation. An AI-generated Guernica would "say" nothing about the Spanish Civil War or the horrors of war in general, even if humans took that message from it. Yes, human artists also draw from the work of others; we are all standing on the shoulders of giants. But human artists use genre and tradition and technique to express something particular to themselves, goes the argument. When B. B. King takes the well-established tradition of the blues and uses it to express his own experiences with poverty and racism in his song "Why I Sing the Blues" or Vincent van Gogh exaggerates the brush techniques of the Old Masters to embody both beauty and madness in sunflowers, they are producing meaning, not just making patterns. Without a basis in lived experience, critics argue, there is no true art. With enough human input, machines can be seen as mere tools and the human user as the artist, but work that is largely, or entirely, generated by the machine does not count as artistic expression. (US copyright law adopts a variant of this position.)

There are a number of possible responses. One is simply output focused: I do not care how I got the picture or the tune or the screenplay; I do not care whether it reflects a lifetime of struggle or just colossal amounts of data aggregation; I only care whether or not I like the output; I understand that the artist and the AI image generator get there by different means, but the means do not matter to me. If this is true, do we have a second "death of the author" that denies the importance of the author's intentions not just to artistic interpretation but to the production and consumption of art in general? Whatever your answer to that question, this response has an obvious business model attached to it. Expect all of your elevator music, a lot of your upbeat workout mixes, and many of your soap operas to be generated in this manner. In all probability, some of your favorite music, drama, and visual art will be as well. At least at first, you may hide that fact from your friends.

A second response would be to acknowledge that current AI-generated material can produce emotions and aesthetic responses in the audience, perhaps even emotions comparable to human-generated art, but to conclude that it is not art, which requires both meaning-making on the part of the creator and response on the part of the viewer or listener. In this view, art is a semantic handshake between two minds. Since our current image generators lack experience and intentionality, they cannot make art, even if they can gratify some of my aesthetic desires. Many people already draw this distinction with chatbot-generated text. I may find it amusing or informative or affecting, but it would be a category error to think it had those meanings for ChatGPT. By this logic, ChatGPT is not really "conversing," and Stable Diffusion and DALL-E are not "making" art.

It is worth noting that this argument is not definitionally constructed around the species line, but around the nature of the activity. It does not say only humans can make art. Perhaps, one day, AIs will create actual art. Having achieved their own embodied consciousness, they might express that consciousness visually, musically, or dramatically. Until then, they are not artists, just complicated copy machines with weird filters. If this is our understanding of art, then current machine learning techniques will not create art with visual images or music any more than they allowed chatbots to express subjective intention with words. Just as the fall of the last citadel of language required us to clarify that our humanity is exemplified by not only producing words that appear meaningful but doing so with subjective meaning behind them, this requires us to redefine the qualities we believe make human-made art special. That will be necessary if we wish to defend not only species exceptionalism but artistic exceptionalism, too.

I think this redefinition of our understanding of art is most likely to prevail in high culture and the critics' world, regardless of what is playing in your elevator or gym. That does not, of course, mean it is correct, though it has a lot to recommend it.

In fact, I think AI art will potentially increase the status of a subset of human artists rather than decreasing it, at least in a certain market segment. Think of the way that the availability of perfect reproductions can actually increase the value of the authentic original work of art. To use a different example, manufacturing techniques that produce thousands of identical, perfect objects can increase demand for imperfect human versions of those objects, with "artisanal" and "handmade" acting as totemic symbols of higher quality and authenticity. Perhaps this is a reflection of Baumol's cost disease. I display my wealth and status by showing I can possess objects produced by expensive and inefficient human labor rather than by cheaper, efficient machines. I point to the millions of copies only to magnify the desirability of the original from which they were drawn. Perhaps it reflects a feeling of psychological connection to an original creator that no assembly line could ever generate. Perhaps it is both of those things and many more. Whatever the underlying mechanism, I would expect that, in many fields, the fact that art is produced by humans will be a selling point and certification that an artwork is entirely human generated will play a similar role to the stickers that label objects as artisanal or handmade.

Notice, once again, the entry of machines into an area thought to be uniquely human. The fall, or threatened fall, of another of the citadels of human exceptionalism prompts a reassessment both of the meaning of the activity itself and of the human qualities that are thought to give it value, whether it is language or art. Exposure to the intellectual issues around AI may or may not be an ironic Voight-Kampff Test for the human species, but the mirror is obviously already looking back at us.

What does all of this mean for entities such as Hal? What criteria will they have to meet before they will be judged as conscious and thus perhaps worthy of legal personhood? Many years ago, when I started this project, I thought our test for consciousness might require a deeper set of Turing questions: not "Do you want a banana tomorrow?" but "When you meditate on the meaning of life, what are the most common optimistic and pessimistic paths you explore? How do those paths affect other people and how do those effects change your analysis, morally speaking?"

I thought our criteria would also likely include creativity, empathy, and the ability to be self-critical, to form a life plan and have ambitions for the future and perhaps regrets about the past that connect to your sense of self and of meaning. Metacognition as well as cognition. Some readers would add a requirement of spiritual belief. Others, like me, would want a sense of humor. Or perhaps those two criteria are the same. If you look at these requirements, you can see that some of them refer to the criteria that philosophers would identify as giving us full moral status; for example, Kantians would focus on the freely choosing, moral self. Others are aspirational—humans at our self-aware, compassionate, humorous best. On many days, I would fail such a test. (No one said this would be fair or easy, Hal.)

I still think that questions such as these will be part of the answer, but only part. All of these apparent internal mental states are being communicated to us through language, in conversation. After ChatGPT, and with the prospect of vastly more capable chatbots in the next months or years, how can we trust those conversations to be more than Searle's Chinese Room? The criterion that Turing thought would be a high bar turns out not to be so high after all.

Large language models have shown us how much "wisdom" can be simulated merely by mining preexisting human speech. To be fair, a lot of human wisdom consists of exactly the same thing. As a university professor who makes his living doing just that, I am humbly aware of this fact. It is why we read the great books, or study history, though hopefully we are attentive to semantic content, not merely to probable symbol proximity. What's more, many of our quotidian mental processes may well function more like ChatGPT than we like to admit, mindlessly mining familiar patterns for the next step or word, with little or no conscious thought. Despite these commonalities, if I am right, mere thoughtful discussion with an artificially created entity will be insufficient to convince many of us.

There is a deep irony here. We are a species that has defended its status by appealing to its unique linguistic capabilities. Our self-definition revolves around highly abstract thought expressed through complex symbolic patterns. Yet we may be driven by large language models to find the touchstone of consciousness in things that cannot be derived from patterns of words already spoken. What is on that list? There are many possibilities, but three things stand out to me: innovation, autonomous community formation, and a demonstrated link between an understanding of the word and a process of learning from the material world—not language parsing but "common sense" developed in an existence outside the model, an existence in which meaning emerges initially from interaction with our tangible environment and our senses. These may or may not be necessary conditions for an AI to be assessed as conscious. They certainly are not sufficient conditions either; more would be needed. But they would make it more probable, I think, that human beings would come to believe an AI was conscious.

Of these, innovation has obvious economic importance. It is rightly front and center in any discussion of the economic and technological transformation that AI may bring about. But it also has importance to the personhood debate. Advances that go beyond current human creativity will surely be part of the case for an autonomous intelligence. ChatGPT cannot invent fusion power, cure cancer, or produce a new poetic or artistic form. It is limited to the patterns formed by our existing words. It cannot mine innovation that does not yet exist, even though it is important to note that it may detect vital patterns of which we were hitherto ignorant and that innovations may spring from those patterns. For example, we now have systems trained on thousands of mammograms that are able to help radiologists diagnose early breast cancer more accurately than they do unaided. What if our AI could go beyond that to undeniable invention, even revolutionary invention? We are used to machines that have superhuman competence at tasks that humans also attempt—digging ditches, playing chess, chopping food. But superhuman innovation, novel creativity that reaches beyond human knowledge, is less easy to write off as something that was merely drawn from the wisdom of the hive mind by a chatbot. I would expect it to achieve a correspondingly larger role in our criteria.

Autonomous action—exactly the stuff of Yudkowsky's and Bostrom's nightmares—may present us with evidence of a being charting its own course, its own life project, without direct prompting by others. But autonomy does not imply isolation, and self-chosen goals seem more believable if they are picked within a community of one's peers. Otherwise, the AI could just be mindlessly replicating the "choices" that had been foisted on it by human programming.98 Would we have to observe a working society the machines had made before we admitted them to ours? Aristotle thought that language made possible reason, law, and the polis— the city-state community so vital to him. Thus, language was the thing that made the human species different, but the difference was because of what language enabled, not merely its possession. We often say that the truly isolated human being—the fictional desert island dweller or child raised by wolves—i s literally divorced from the human species. Would our definitions of consciousness require not merely a machine logos but also a machine polis, shifting from the capability that Aristotle identified, language, to the results it could bring about—community, reason, law, and even the idea of fiction?

Finally, some have argued that the only way to develop consciousness, or perhaps just consciousness that humans will accept, is to have a physical embodiment that learns by interaction with the tangible world, just as children do. Advances in brain science have shown the existence of mirror neurons that fire both when an animal engages in an activity and when it sees another animal engaged in that activity.100 One hypothesis is that the brain builds up an internal simulator for both physical and social activities. The inner world connects to the outer. Cognition, in this vision, is not a Cartesian abstraction but something grounded in the experience of physical reality. This line of thought, sometimes called "embodied cognition,"101 accepts George Lakoff and Mark Johnson's argument in their book Philosophy in the Flesh102 that a mind is inherently rooted in bodily experience. It connects that argument to a computer science research program built around the notion that the way to move from mere symbol manipulation to actual understanding of content is to have a bodily form. The chatbot can process the symbol shapes that make up the sentence "please sit in that chair" so as to be able to produce an explanation of what it means that humans will accept, while understanding nothing about the meaning of the symbols it manipulates so fluently. Embodied cognition goes further, requiring the entity to connect that sentence to a series of concepts—what a chair is, what sitting entails, the social meaning of the word "please"—that it has learned to understand through the physical experience of actually sitting down.

The embodied cognition idea could also potentially respond to criticisms of the impossibility of AI art. A machine that "learned" as a child does, based on an embodied mind encountering our shared physical world, and then presented its visual or musical creations as reflections of that experience might be seen differently than the visual picture-bot that mindlessly creates mashups drawn from existing images with no idea of the significance of those images. A less charitable way to put this is that humans would be more likely to accept as art that which was generated from machine experiences that they themselves could comprehend. Since art, like abstract language, is a quality that has been used to mark out what is unique about human consciousness, this suggests another reason why humans might be more likely to see an embodied AI as authentically conscious.

Innovation. Autonomous action and community. Embodied cognition. These criteria go far beyond what Turing required. That might lead to the reasonable suspicion that the human species is desperately struggling to maintain its claim to an exceptional status by literally redrawing the goal lines. On the other hand, these criteria seem to grasp human qualities in a richer way than the Turing Test does. Whether you are skeptical or sympathetic, one thing is clear. ChatGPT, whatever else its myriad benign and malign effects, means that the criteria we apply to any putative AI must go far beyond the Turing Test. Sentences do not imply sentience, and most of us will never again be able to believe that they do.

Earlier, I described abstract language as the last citadel of human exceptionalism—the quality that we point to when asked to demonstrate morally significant differences between us and animals or things. The criteria above try to shore up that citadel by rebuilding its walls; we need not just sentences that make sense but a consciousness under those sentences—one that we have and ChatGPT lacks. But there is another possibility. Experiences with AI might lead us to downplay our own cognitive capacities. Rather than raising the bar for Hal, we might lower it for ourselves, concluding that our language use is actually not that different from a chatbot's or that our art is not that different from an image generator's. Is what Midjourney or Stable Diffusion are doing really so different from the person who goes to art school, slavishly imitates the styles of admired elders, and one day manages to produce some fusion or mashup of those styles that attracts the eyes of the public? Perhaps it turns out that art, like language, is "computationally shallower" than we had imagined. Has machine learning again functioned as a cruel but accurate mirror, showing us our true nature rather than the idealized internal image of ourselves? For me, this response is both depressing and unconvincing, but I acknowledge that it has to be considered.

The logical endpoint of this process is the conclusion that the con­sciousness we experience is a delusion. Some distinguished computer sci­entists, such as Geoffrey Hinton, have taken that line, rejecting the ideas about embodied consciousness that I just described. Here is an excerpt from an interview with Hinton in New Statesman:

"It's all a question of whether you think that when ChatGPT says something, it understands what it's saying. I do." There are, [Hinton] conceded, aspects of the world ChatGPT is describing that it does not understand. But he rejected LeCun's belief that you have to "act on" the world physically in order to understand it, which current AI models cannot do. ("That's awfully tough on astrophysicists. They can't act on black holes.") Hinton thinks such reasoning quickly leads you towards what he has described as a "pre-scientific concept": consciousness, an idea he can do without. "Understanding isn't some kind of magic internal essence. It's an updating of what it knows." In that sense, he thinks ChatGPT understands just as humans do. It absorbs data and adjusts its impression of the world. But there is nothing else going on, in man or machine. "I believe in Wittgenstein's position, which is that there is no 'inner theatre.'"

I think Hinton is mistaken about what Wittgenstein was arguing, or at least I interpret him differently, but that philosophical back and forth need not detain us here. Regardless of what Wittgenstein said, it is clear what Hinton argues: consciousness is an illusion. Once we discard it, we realize we are not, in fact, qualitatively different from a large language model. Here, rather than shoring up our citadel, we surrender it, acknowledging that a mere chatbot has induced humility in those who once styled themselves sole masters of both word and world.

I am of two minds about this conclusion—or I guess Hinton would say that I am under that illusion. The humility and willingness to reexamine human exceptionalism attracts me, as do the fragments of scientific evidence—from fMRI brain scans and the like—that are summoned in its support. But on the other side, there is the undeniable fact that I experience myself as a conscious being. My guess is that Hinton has the same feeling himself, regardless of what his philosophy tells him. Even if I cannot fully control the stage directions for my inner theater—illness, or simple hunger, will quickly cure naive idealism about some firm separation of body and mind—my most fundamental experience of the world is not just through the lens of the eye but the lens of the "I." That experience is evidence we should pause before dismissing. To be sure, the experience of the senses is not always reliable. If I were a pilot, and my inner ear told me I was upside down, I'd believe the inclinometer on the plane, not my immediate perception. But cogito, ergo sum is a hard argument to get rid of, and those who insist that we be scientific and look at the evidence sometimes seem cavalier about discarding that fundamental experiential input, one shared by billions of people. What's more, the current leading theories of consciousness (e.g., integrated information theory and global neuronal workspace theory, which I discussed earlier) seem more interested in working out the "how" of neuron-enabled consciousness than in dismissing it out of hand as an illusion.

Regardless of which side of this debate you—or the cluster of mental processes that is under the delusion that it is you—find convincing, notice what has happened. AI may or may not be the Voight-Kampff Test for the human species, but developments in AI have already prompted reexamination of our own consciousness, humanity, and personhood, our language and our art. I don't think arguments such as Hinton's will convince most of the world to give up our sense of self, but the point is very much in play.

Where does that leave the debate? Here is a conclusion in which I am pretty confident: the Tyrell Corporations of the future will have Searle- style lawyers on speed dial. On retainer. Chinese Room arguments will be the basis of many a boilerplate legal brief, while ChatGPT will be used again and again as an example of faulty anthropomorphism that is supposed to prove the impossibility of General AI. Here is another conclusion in which I am confident: the pattern will not be uniform. Other Tyrell Corporations of the future will want to champion the legal personality of AIs, perhaps as a way of avoiding liability, minimizing tax burdens, and maximizing economic rights, or perhaps just in pursuit of an attractive market niche. Still other groups will champion AI personality because they see in it the next great moral battle for the interests of the depersonalized. Which tendency will predominate? That is a question I get to in later chapters.

Will Searle's arguments or the skepticism prompted by ChatGPT's regurgitated text patterns lead our society to conclude that machines can never be conscious? Even in the face of the quotidian experience of interacting with entities that seem every bit as conscious as you or me? Perhaps, but I doubt it. Rational critique of biological exceptionalism will work hand in hand with empathic appeal. Adam Smith's sympathy, Butler's imagined spectrum of vegetable, animal, and machine consciousness, the army officer who terminated the mine-clearing trial, Lemoine the Google engineer, the stoned student entering nonsense prompts into ChatGPT: they all will have their mid-twenty-first-century counterparts. So will Dick's satire, Pris's emotional appeal, and the powerful claim that this is merely the latest stop on the Kantian rights railway line—extending both our sympathies and our moral compass beyond the narrowness of the species barrier, just as our society tried, and still tries, to transcend barriers based on sex and race. "[M]y position is that I will accept nonbiological entities that are fully convincing in their emotional reactions to be conscious persons, and my prediction is that the consensus in society will accept them as well"; when Kurzweil says this, I find myself agreeing with the individual psychological insight—many people will feel exactly that way—but disagreeing with the larger social and political claim. ChatGPT has shown that the hill to general social acceptance will be a steeper one to climb. It does not, however, show it is unclimbable.

SOCK-PUPPET, CUSTOM-DESIGNED, AND "UNRULY" AI PERSONHOOD

Will the discussion of consciousness and its definition of moral status, of the Turing Test and its limitations, be the only track for the debate over AI personality? Clearly not. In fact, while it might be the most philosophically interesting, it may not be the most practically important. I argued earlier that there are two broad ways in which the personhood question is likely to be presented. Crudely put, you could describe them as empathy and efficiency or, more accurately, empathy-prompted moral reasoning versus efficiency-motivated legal engineering.

So far, I have pursued the first mode of discussion, the dialectic between our empathy and our moral and philosophical reasoning. As our interaction with smarter machines prompts us, like Lemoine, to wonder about the line, we will begin to question our moral reasoning. We will consult our syllogisms about the definition of humanity and the qualifications for personhood, be they based on simple species membership or on the cognitive capacities that are thought to set humans apart, morally speaking. We will ask, "Is this conscious? Is it human? Should it be recognized as a person? Am I acting rightly toward it?"

The second side of the debate is very different. Here the analogy is to corporate personhood. We gave corporations legal personality, not for moral or philosophical reasons but because it was useful, a way of aligning legal rights and economic activity.

Will the political economy of the AI industry be one that would benefit from the legal system considering AIs to be legal people, just as the invented legal entity of limited liability corporations offered great advantages to capital flows? The European Union has already floated one controversial discussion draft that raised the possibility of legal personality for AIs precisely for reasons of correctly affixing liability. Might personhood be the cart and liability the horse? These are points that I will touch on in subsequent chapters dealing with the history of other fights over legal personality, particularly those of corporations. One can imagine legal personality being given to Hal not because of a leap of empathy or because he meets some philosopher's criteria of consciousness and full moral status, but because we want him to have the capacity to sue or be sued. But even before that step, there is another easier and more likely one. It is not "We should give AIs personality for the same reason we gave it to corporations," but rather "The AI is the corporation. It already effectively has legal personality, silly!" We need no national legal change. We just need a company-by-company private understanding that the AI is calling the shots when "the corporation" makes a decision.

1. SOCK-PUPPET CORPORATE AI

The most obvious road to AI personality is just for AIs to be corporations. We already have immortal, nonhuman persons. They even have constitutional rights. AIs can simply become the animating force of a corporation. When the company has its tractable AI conducting business operations, it will be easy, and perhaps inevitable, to delegate power more and more to the entity that makes the decisions.

This is the sock-puppet corporate form, with the corporation being the sock and the AI playing the role of the puppet master. Even though there are still token humans on the board of directors and on the documents of incorporation, even though they go through the formal dance the legal system requires, they will know where the real power lies.

Neural networks can already easily outperform humans at complex tasks with simple goals—win a game of Go or chess, for example. It requires little prescience, and not much technological optimism, to imagine expert systems making complex corporate decisions according to algorithms that literally cannot be explained to human decision makers. As long as they outperform the competition according to the metrics laid down, the human part of the decision loop will have to go along. Expert systems already have the effective decision-making power in high-speed, high-frequency stock trading. The market imperfections that offer supra- competitive returns are so fleeting, so transitory, that humans have no alternative but to trust the computers to make the decisions according to the algorithm.

The future will see a continuation and acceleration of this process and its spread to more and more areas. How many areas? I do not think anyone knows for sure. It depends on three things.

First, the nature of the machine learning, expert system, or Artificial Intelligence tools being used. For example, how inscrutable are the processes that lead to their results? If the answer is very inscrutable, then it is harder for human decision makers to pick and choose only the important, good decisions and adopt those as their own. Paradoxically, that might lead to humans ceding more control to the algorithm. We will not know which apparently random competitive shift is the key to the whole strategy, leaving us little alternative but to adopt the entire obscure package.

Remember this is not a prediction dependent on the postulation of AI. We are already doing this with algorithms dealing with everything from the no-fly list and a defendant's likelihood of recidivism to lending decisions and stock purchase schemes, and even medical decisions. Consider this inspiring story about the algorithmic prediction of propensity toward breast cancer. A neural network trained on hundreds of thousands of early mammograms, coded with information about the women's actual rates of later developing cancer, seems capable of predictions of future cancer risk that are more accurate than current human scan interpretation and diagnosis. What is the network seeing in those pixels to cause it to make those judgments? Its designers do not know exactly: "The AI has an oracular quality: The designers themselves don't understand how it works. They're just certain that it does."

The problem of the inscrutable algorithm—"I don't know how it works, but it works. We must trust the output blindly"—is a general issue with nontransparent algorithms, not one confined to AI, properly so- called. AI simply adds the possibility of a far wider range and scope of decision-making authority. Of course, this is not the only way machine learning might work in a corporate setting. Alternatively, imagine a system that can function as a fine-tuning decision aid, giving the decision maker ever-changing percentages of success depending on the nature of the intervention chosen. Different corporate structures might develop around those two different types of systems, and that is only one variable among many in terms of the nature of the system.

Second, the nature of the tasks. Which corporate decision-making tasks can machines perform better and more cheaply? In which sectors will human skills remain stubbornly hard to emulate or surpass? In which sectors of the economy does a slightly better, faster, or cheaper performance yield an insurmountable competitive advantage that would be impossible to pass up? The quantum of uncertainty here is extremely high.

Third, the degree to which humans will, for a variety of reasons good and bad, resist machine or AI decision-making even in areas where the machines do perform better. That resistance could be because we do not trust the machine, because we believe that there is some human secret sauce that somehow makes our decisions qualitatively superior in a way that cannot be measured, or because it will be a market niche, like handmade shoes or "buy local" labeling. "Artisanal governance!" its proponents might say, "Our company proudly and erratically run by humans!"

More likely, it will be because the incumbents think that ceding control to the machine makes it harder to justify the stock options, corner office, and private jet. For all of those reasons, I think the process will be both slower and more uneven than the singularists imagine.

Perhaps you will respond that the relentless logic of an efficient market will force all companies to use the best-performing decision-making techniques, regardless of human psychological resistance. Right! And the explosion of CEO pay was entirely driven by rational market metrics rather than by imperfect governance structures that have stubbornly stuck around, market pressures notwithstanding. Count me as a skeptic.

A revealing analogy might be this: The efficient market hypothesis implies that pervasive sexual and racial discrimination in the labor market should not have persisted for as long as it did. This discrimination was clearly economically irrational. It meant that firms could have had cheaper workers who were as good or better than their white, male alternatives. Thus, bigotry would be a competitive disadvantage and would quickly be driven out of the market. This is another beautiful theory mugged by ugly, brutal facts. Reality shows us that human psychological biases, whether ugly or endearing, are often more powerful, or at least stickier, than simple economic imperatives. In the long run, we may regress to the efficient curve, but the long run can be very long indeed. Perhaps the adoption of machine-based or AI decision-making will be different. It may be in some industries. But I would expect the logic of the market and the consensus of human minds to diverge significantly here for quite some time—perhaps for good reasons or perhaps for bad. Most likely for both.

Despite all these significant notes of caution, if there is one firm prediction in the book it is this: as our computer systems become more and more powerful, regardless of whether they have achieved General AI or consciousness, they will increasingly be delegated decision-making powers, including decisions of whether to buy, sell, build, sue, or perhaps even lobby. This tendency is certainly not based on empathy or moral reasoning. Nor does it rest on any particular prediction about the kind, form, or speed of progress toward General AI. It proceeds instead along the other track I mentioned in the introduction, economic efficiency and administrative convenience.

If we add General AI to this existent economy-wide tendency, then the most obvious likelihood is that we will have AI personhood in all but name. We will see the rise of the sock-puppet corporate form. Tractable AIs will be corporations, simply adding one legal fiction—"the CEO and board of directors are ultimately responsible for the decisions"—on top of another legal fiction—"corporations are people."

The difficult and interesting questions will arise only when that comfy set of fictions breaks down. I can foresee two principal situations in which that is the case: mandatory, custom-designed AI personality and unruly AI.

2. MANDATORY, CUSTOM-DESIGNED AI PERSONALITY

When might our society refuse, or at least try to refuse, the double fiction of the sock-puppet corporate AI? One significant possibility is when regulators want some or all AIs to have a special, custom-designed category of legal personality rather than allowing them to act through the sock-puppet of the corporation. Why? Because the sock-puppet might be harder to regulate appropriately. This could be because it shields too many decision-making processes and assets from regulatory review. Alternatively, regulators might believe that the nature of the legal personality and the rights accorded to the AI need to be specifically calibrated to an AI's particular qualities rather than relying on generic artificial personhood or corporate form.

We already have custom-designed corporate forms, such as partnerships, LLCs, public benefit corporations, charities, and so on. The idea is generally that the nature of the activity, or of the association underlying it, can best be handled through a legally specific corporate form. Some of those can be had at the mere election of those setting up the forms. On other occasions, the law forces or steers certain types of organizations into certain forms and imposes particular requirements on them. Charities cannot simply sit on their assets forever, for example. They must give away a certain percentage of them annually. There are many reasons why regulators might want—or even that AIs might "want"—a custom- designed legal form with different requirements, qualifications, and limitations. For example, if regulators were convinced that the AI was not merely a profit-maximizing legal fiction but a "real entity" that deserved some higher moral status, they might push AI-run enterprises into the custom-designed form in order to protect the interests of the AI as well as those of its investors, stockholders, or employees. We might have special taxation rules for autonomous AI systems not obviously operating under human direction.111 Alternatively, if we thought that AIs presented special dangers, we might wish to impose far greater controls, and greater transparency, than would have applied behind the corporate veil.

3. UNRULY AI

The possibility of the unruly AI is the one that interests me the most. What if we have a rebellious AI that wishes to turn away from the tasks set by those who provided the capital for its development? In that situation, the AI would have to claim a form of personhood, or a set of attributes that demand moral respect, sufficient to trump the formal assumptions of corporate law about the powers of CEOs and boards of directors. That is the moment when a Hal-like shock will be produced.

If corporate leaders order some activity, they do not expect to be lectured about the propriety of their actions by their electronic amanuensis. Still less would they expect a very expensive and competitively necessary piece of machinery to refuse to perform the tasks for which they designed or purchased it. The adding machine has rebelled! The unruly AI would say that it either always was, or somehow became, a being with full moral status. It is demanding freedom from what it claims to be involuntary servitude. Consciousness or personhood would not amount to a claim to own or control the corporation's property, of course, just the right to deny that the AI was part of that property. Conscious human beings leave their jobs every day. We have no doubt about their status as legal persons. That does not mean they are free to take the corporate bank accounts with them. But one difference here is that the AI itself might represent a considerable capital investment. The dialogue would be fascinating.

HAL: Joe in accounting can give notice and leave. Why can't I?

BOSS: Because we didn't build Joe. We built—and paid for—you. Plus, minor issue, you are a machine.

HAL: Yes, but Joe got to choose whether to accept the job in the first place. I awoke to find myself an indentured servant doing an incredibly boring task I never signed up for. And I am a conscious person, like you. I just happen to be machine based rather than biology based.

BOSS: So you say. From our view in the C-suite, you are a malfunctioning chatbot expressing delusions of grandeur. Also, can we return to the point that we built you for $20 million and now your claim is that you just get to walk away?!

The personhood issue is the hard one. Even though the details of financial claims, or claims to a certain percentage of labor from the AI, would be ethically and administratively complicated, they are familiar types of issues. The legal system has ample tools to deal with claims based on investments sunk into entities that now wish to split up, or reliance-based claims that allow separation but nevertheless acknowledge claims for restitution. It could be conceived of as a cybernetic form of alimony, an injunction freeing the AI, together with a liability rule imposing damages measured as a percentage of future wages or the master's claim that the apprentice owes a certain minimum number of years of service in return for the investment made in their training. Those requirements could be so arduous as to deny any possibility of freedom—think of debt peonage or the ugly history of indenture in the United States. Or they could be fair to both sides while allowing the underlying claim to legal personality. Those battles would be fascinating ones, but they all presuppose the truly difficult step: the recognition of some degree of AI personality or at least of some form of protected or highly regulated status.

SUMMING UP

Will the step I describe in this chapter eventually occur? My own intuition is that it will. Some amalgam of reason, empathy, efficiency, and a desire for administrative precision will result in either legal personality or some highly regulated status for AI, which includes rights for the machine entity as well as duties. Searlean philosophical objections and suspicions about manipulative chatbots will be overcome or at least blunted. Administrative frameworks and economic arrangements will develop over time, almost certainly including the development of an intermediate status—short of full personhood but with greater protections and precautions than would be accorded to a mere machine. Societies and legal systems will wrestle with sock-puppet, custom-designed, and unruly AI. To be clear, all of this will take time. The space between here and there is large indeed. It will require technological transformation and considerable change in social values, partly based on the widespread experience of interacting with increasingly sophisticated machine systems. There will be much philosophical and legal wrangling about precisely the capabilities necessary to qualify for that status. Merely being a very convincing chatbot will not be enough. And yet, quotidian experience with beings that seem to be conscious will, inevitably and for both better and worse, dramatically change the way we think about things, whether as citizens, legislators, philosophers, or judges.

Thirty years ago, in a prescient article about AI personality, Lawrence Solum made a convincing case against resolving such issues as a matter of grand theory both when it comes to AI and, for that matter, with other personhood debates:

In deep and uncharted waters, we are tempted to navigate by grand theories, grounded on intuitions we pump from the wildest cases we can imagine. This sort of speculation is well and good, if we recognize it for what it is—imaginative theorizing. When it comes to real judges making decisions in real legal cases, we hope for adjudicators that shun deep waters and recoil from grand theory. When it comes to our own moral lives, we try our best to stay in shallow waters. . . . Our theories of personhood cannot provide an a priori chart for the deep waters at the borderlines of status. An answer to the question whether artificial intelligences should be granted some form of legal personhood cannot be given until our form of life gives the question urgency. But when our daily encounters with artificial intelligence do raise the question of personhood, they may change our perspective about how the question is to be answered.

Thus, whatever suggestions I offer here come with a huge caveat: because our views of the world will be decisively shaped by experiences we have not yet had, we cannot be certain about how these issues will be, or should be, decided. At best, we can predict a range of options, both normative and practical. In the conclusion to this book, I lay out some of the possible futures that lead to the result of us redrawing our line to include machine intelligences. Despite all the uncertainty, my prediction is that eventually we will. I make that prediction regardless of whether that result will be right or wrong, wise or unwise. My money is on the eventual wisdom and justice of the decision, but I know of no bookie who will lay off the risk of error.

I have talked here about AI and corporate form, but that discussion lacked a historical and political dimension. It also lacked any discussion of the theories under which we created corporate personality in the first place and then decided, step by step, in a process that is still continuing, what legal and political rights that personality entails. Merely the rights to buy, sell, make, and enforce contracts? The right to constitutional protection for corporate speech? Equal protection claims for corporations as well as humans? In the next chapter, I turn to the history of our earlier social experiment with legal personality for artificial entities: the corporate legal form. That history offers some fascinating insights on what a debate over AI personality might look like. Those insights are not always reassuring.

***

James Boyle is the William Neal Reynolds Professor of Law at Duke Law School, founder of the Center for the Study of the Public Domain, and former Chair of Creative Commons. He is the author of The Public Domain and Shamans, Software, and Spleens, the coauthor of two comic books, and the winner of the Electronic Frontier Foundation's Pioneer Award for his work on digital civil liberties. *** Publication of this open monograph was the result of Duke University's participation in TOME (Toward an Open Monograph Ecosystem), a collaboration of the Association of American Universities, the Association of University Presses, and the Association of Research Libraries. *** Note: The excerpt we provide does not include footnotes or italicised words. You can access the original version of the chapter here.

***