The Coming Wave of AI Regulation

A recent Twitter thread from a (now ex-) Anthropic and OpenAI researcher has gone viral, eliciting reactions from a diversity of crowds. In his posts, the researcher explains that he has decided to quit his job because the frontier labs are irresponsibly striving to create a “superintelligence” that could potentially kill us all. One potential solution, he claims, is coordination between the frontrunners; so-called pacing agreements that would regulate the rate of innovation. Since the thread was posted, I’ve seen many politicians advocating, in various terms, for more regulation. I use the term “regulation” generically to refer to legal rules that regulate conduct, including legislation, administrative regulation, etc.

Without more information, I’m skeptical of such responses. Two market leaders coordinating as such would likely help suppress competition and be illegal under antitrust laws. Of course, if humanity’s on the line, we shouldn’t care about competition laws. But the evidentiary bar to justify setting aside normal concerns should be high.

Government intervention should also be well informed. I sense there’s a feeling that regulation of social media / search companies was too little and too late, and people are eager to avoid that mistake with the new AI tech giants. I think this instinct is correct, but hazardous without more information. Premature or overly broad regulation could be over-inclusive, entrench the current winners, all at the cost of innovation / the consumer / society.

With this said, I want to make two things clear:

(1) Regulation could be necessary and appropriate for well defined or active threats. I’m not sure exactly where to draw the line between well vs. ill defined threats, but it seems clear that some threats are on the well-defined side of the line (e.g. AI-assisted hacking, abusive imagery) and some on the other (e.g. harms from a superintelligence).

(2) That said, there’s also a place for information-forcing regulation designed to improve our collective understanding of speculative and less well defined threats (eg consequences of RSI; expected capabilities). These kinds of regulations are not costless for firms but are initially preferable to restrictions that use this information to narrowly tailor future “substantive” regulation. If after analyzing the relevant information / data, disinterested third parties conclude that we are on pace to build a humanity killing AI, then we should intervene aggressively. For certain types of threats (eg bio), that intervention might not even be around AI itself. But to make good interventions, we need more information. And right now it feels like the public is flying somewhat blind.

All of the preceding basically boils down to “how much information do we need about an ongoing or potential harm to justify intervention?” It’s possible I’m under-informed and projecting that onto the general public. But the fact that, to my knowledge, we haven’t even yet had a proper investigation of the OpenAI-HF incident makes me question our ability to effectively assess threats originating from “misaligned” agents.


Our appetite for regulating AI seems to be growing. I expect datacenters and AI will feature regularly as we approach the midterms, and for that enthusiasm to translate into passed legislation and promulgated regulation. I just hope these new rules have a firmer basis than tweets and voluntary disclosures.

Final point: I’m not suggesting this particular researcher or these firms more generally intend anti competitive results. Instead, I think pre-mature regulation created without the private information that elucidates these harms can have anti competitive effects.