The Unsettling Lesson of the OpenAI Mess

Science fiction writers and artificial intelligence researchers have long feared the machine you can’t turn off. The story goes something like this: A powerful A.I. is developed. Its designers are thrilled, then unsettled, then terrified. They go to pull the plug, only to learn the A.I. has copied its code elsewhere, perhaps everywhere.

Keep that story in mind for a moment.

Two signal events have happened in A.I. in recent weeks. One of them you’ve heard about. The nonprofit that governs OpenAI, the makers of ChatGPT, fired Sam Altman, the company’s chief executive. The decision was unexpected and largely unexplained. “Mr. Altman’s departure follows a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities,” read a cryptic statement.

Many assumed Altman had lied to the board about OpenAI’s finances or safety data. But Brad Lightcap, an OpenAI executive, told employees that no such breach had occurred. “We can say definitively that the board’s decision was not made in response to malfeasance or anything related to our financial, business, safety or security/privacy practices,” he wrote. “This was a breakdown in communication between Sam and the board.”

At the heart of OpenAI is — or perhaps was — a mission. A.I. was too powerful a technology to be controlled by profit-seeking corporations or power-seeking states. It needed to be developed by an organization, as Open AI’s charter puts it, “acting in the best interests of humanity.” OpenAI was built to be that organization. It began life as a nonprofit. When it became clear that developing A.I. would require tens or hundreds of billions of dollars, it rebuilt itself around a dual structure, where the nonprofit — controlled by a board chosen for its commitment to OpenAI’s founding mission — would govern the for-profit, which would raise the money and commercialize the A.I. applications necessary to finance the mission.

In my reporting, the best explanation I’ve heard for the board’s decision, as poorly communicated as it was, is that it fired Altman for the precise reason it said it fired Altman. He was acting and communicating with the board in a way that made it hard for it to “exercise its responsibilities” — that is to say, control the for-profit.

Part of this was communication; the implication is that the board felt Altman was slow or opaque or misleading in what he told them and when. But part of it was the pace of commercialization, the contracts being signed with partners, the promises being made to employees and the products being pushed into development. OpenAI is supposed to develop A.I. to serve humanity, not to help Microsoft capture market share against Google. And so the board tried to do the thing that it was designed to do: rein in the for-profit, in this case by firing Altman. Warned that firing Altman could destroy OpenAI, Helen Toner, one of the board’s members, replied, “That would actually be consistent with the mission.”

OpenAI had always told its investors that something like this might happen. “It would be wise to view any investment in OpenAI Global LLC in the spirit of a donation,” it warned. The company then went out and raised tens of billions of dollars from people who did not see their investments as donations. That certainly describes Microsoft, which invested more than $13 billion in OpenAI, owns 49 percent of the for-profit, put OpenAI’s technologies at the center of its product road map and has never described its strategy as philanthropic.

Satya Nadella, Microsoft’s chief executive, didn’t have the power to stop OpenAI’s board from firing Altman, but he did have the power to render the action hollow. He announced a new advanced innovations division, to be led by Altman (and the OpenAI president Greg Brockman, who resigned in solidarity with Altman) and staffed by any OpenAI employees who’d like to join. More than 90 percent of OpenAI employees — including, weirdly, Ilya Sutskever, the OpenAI co-founder who’d not only helped the board oust Altman but also had informed Altman that he was fired — then signed a letter threatening to quit unless Altman was reinstated and the board resigned. By Wednesday, Altman was back as chief executive of OpenAI and all but one of the board members had resigned.

I don’t know whether the board was right to fire Altman. It certainly has not made a public case that would justify the decision. But the nonprofit board was at the center of OpenAI’s structure for a reason. It was supposed to be able to push the off button. But there is no off button. The for-profit proved it can just reconstitute itself elsewhere. And don’t forget: There’s still Google’s A.I. division and Meta’s A.I. division and Anthropic and Inflection and many others who’ve built large language models similar to GPT-4 and are yoking them to business models similar to OpenAI’s. Capitalism is itself a kind of artificial intelligence, and it’s far further along than anything the computer scientists have yet coded. In that sense, it copied OpenAI’s code long ago.

Ensuring that A.I. serves humanity was always a job too important to be left to corporations, no matter their internal structures. That’s the job of governments, at least in theory. And so the second major A.I. event of the last few weeks was less riveting, but perhaps more consequential: On Oct. 30, the Biden administration released a major executive order “On the Safe, Secure and Trustworthy Development and Use of Artificial Intelligence.”

It’s a sprawling, thoughtful framework that defies a simple summary (though if you’d like to dig in, the A.I. writer Zvi Mowshowitz’s analysis, and roundup of reactions, are both excellent). Broadly speaking, though, I’d describe this as an effort not to regulate A.I. but to lay out the infrastructure, definitions and concerns that will eventually be used to regulate A.I.

It makes clear that the government will have a particular interest in regulating and testing A.I. models that cross particular thresholds of complexity and computing power. This has been a central demand of A.I. safety researchers, who fear the potential civilizational consequences of superintelligent systems, and I’m glad to see the Biden administration listening to them, rather than trying to regulate A.I. solely on the basis of how a given model is used. Elsewhere, the order signals that the government will eventually demand that all A.I. content be identifiable as A.I. content through some sort of digital watermarking, which I think is wise. It includes important sections on hardening cybersecurity for powerful A.I. models and tracking the materials that could be used to build various kinds of biological weapons, which is one of the most frightening ways A.I. could be used for destructive ends.

For now, the order mostly calls for reports and analyses and consultations. But all of that is necessary to eventually build a working regulatory structure. Even so, this quite cautious early initiative met outrage among many in the Silicon Valley venture-capital class who accused the government of, among other things, attempting to “ban math,” a reference to the enhanced scrutiny of more complex systems. Two weeks later, Britain announced that it would not regulate A.I. at all in the short term, preferring instead to maintain a “pro-innovation approach.” The European Union’s proposed regulations may stall on concerns from France, Germany and Italy, all of whom worry that the scrutiny of more powerful systems will simply mean those systems are developed elsewhere.

Let’s say that U.S. regulators saw something, at some point, that persuaded them they needed to crack down, hard, on the biggest models. That’s always going to be a judgment call: If you’re regulating something that can do terrible harm before it does terrible harm, you are probably regulating it when the terrible harm remains theoretical. Maybe the harm you fear will never happen. The people who stand to lose money from your regulations will be very active in making that case.

Or maybe it’s not even that dramatic. Regulators begin to see emergent capabilities and unexpected behaviors that convince them these systems are advancing too fast, that they are beyond our capability to understand, and the pace of progress needs to slow to allow more time for testing and research. Worse, they see this in proprietary data, which they can’t release to the public. And even if they could release that data, it would still be a judgment call. To act early is to act without certainty.

Would that look to the rest of the world like a wise approach they should copy? Or would it look like an opportunity to snatch the lead in A.I. from the United States? It is not hard to imagine Britain, China or Japan, or smaller players like Israel, Saudi Arabia or Estonia, seeing opportunity in our caution and luring A.I. firms with promises of a more pro-innovation approach. If it seems fanciful to imagine whole companies setting up shop overseas in search of looser regulations, remember that FTX, under Sam Bankman-Fried, really did move its headquarters and staff to the Bahamas for exactly that reason. We saw how that worked out.

I don’t mean to be too pessimistic. If A.I. develops as most technologies develop — in an incremental fashion, so regulators and companies and legal systems can keep up (or at least catch up) — then we’re on a good track. Over the past year, I’ve been cheered by how seriously governments, technologists and the media are taking both the possibilities and the pitfalls of A.I. But if the capabilities of these systems continue to rise exponentially, as many inside the industry believe they will, then nothing I’ve seen in recent weeks makes me think we’ll be able to shut the systems down if they begin to slip out of our control. There is no off switch.

The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: [email protected].

Follow The New York Times Opinion section on Facebook, Twitter (@NYTopinion) and Instagram.