Sometimes, a piece of writing goes so viral that it becomes an event—a Current Thing—in and of itself. One of the most successful recent instances of this is AI 2027: a detailed scenario outlining how AI might doom humanity within a few short years, as accelerating capabilities intertwine with US-China competition, recursive self-improvement, deceptive misalignment, and job loss. Reading it, or at least the summary, is worth doing to understand AI risk debates.
There are two big buckets of discourse about AI 2027:
First is people debating the predictions themselves—How plausible is an intelligence explosion? Won’t adoption lag slow down the risks? What about defensive superintelligence? I don’t get into this much here, and believe that Daniel and Eli have already adjusted their own timelines quite a bit. (This makes me wish the AI 2027 site had a live sidebar ticker showing every new piece of information that updates their thinking, e.g. the METR study about Cursor decreasing dev productivity.)
The second critiques scenario forecasting as a medium.
and argue that AI 2027 obscures human leverage and uses the authors’ credentials to make doom seem inevitable. concurs, saying that forecasts (like expert surveys) conflate empiricism with advocacy—i.e. that scenarios and surveys are always framed to advance the framers’ goals. And user titotal on LessWrong suggests that AI 2027 created a dangerous illusion of certainty given the errors in their model.1
Personally, I’m in the lonely camp of being skeptical about many of AI 2027’s predictions, but appreciative of the format and conversation it sparked. When Jessica and Saffron compare AI 2027 to Kim Stanley Robinson’s The Ministry for the Future, I think: The Ministry for the Future was awesome, and I’m glad it exists! There are plenty of others writing policy reports and op-eds; we need new styles to shock people into thinking in new ways, and to consider a broader-than-usual range of possible outcomes (e.g. I loved this delightful AGI parable from
).I also think it takes real guts to put out predictions that can be so concretely disproven: putting dates on predictions requires skin in the game. The authors will be clowned on when they inevitably get stuff wrong. That suggests the AI 2027 authors really believe in their scenario, rather than doing weird wish fulfillment as some critics say (like I doubt they want us all to die).
Therefore, I invited AI 2027 author
on the podcast to discuss his team’s approach to creating AI 2027, answers to common critiques of forecasting (e.g. is it just bad sci-fi), and why he thinks writing scenarios can improve your thinking.I’d also like to host some kind of amateur AI scenario-writing workshop—a half-day in SF, with folks from a range of backgrounds. I’d love to see more pluralistic visions of what our AI future might look like, and think scenario-writing will be a good discursive tool. If such an event might interest you (or to see more details), fill out this form!
Full transcript
Jasmine Sun (00:00)
Today's guest is
, author of the bombshell essay AI 2027 and former governance researcher at OpenAI.Daniel, along with his co-authors,
and Scott Alexander, have done a number of podcasts and blog posts about their specific predictions. I recommend listening to the Dwarkesh episode (more weedsy) or Ross Douthat episode (less weedsy) if you want that.For today's conversation, we'll talk less about the predictions and more about the value of scenario forecasting as a method for truth-seeking and advocacy. Welcome!
Daniel Kokotajlo (01:31)
Thank you so much for that warm intro, Jasmine. I'm super excited to talk about those questions. I've done a bunch of podcasts about AI 2027 and my background, but we are genuinely hoping to inspire lots of other people to write scenarios about the future of AI.
Jasmine Sun (01:56)
What was the first scenario forecast that you ever wrote?
Daniel Kokotajlo (02:06)
Way back in the day, in 2019, I had just left grad school and began a job at AI Impacts, which was a small think tank that tries to forecast the future of AI. We did a vignettes workshop. We tried to have scenarios depicting what the future might be like.
Jasmine Sun (02:39)
Is that What 2026 Looks Like, or is that an even earlier one?
Daniel Kokotajlo (02:42)
That was two years later. I wrote What 2026 Looks Like in 2021.
Wait a minute, my memory is wrong. I think that What 2026 Looks Like might have evolved out of the AI Impacts workshop. It must have started at that workshop, and then I later expanded it and turned it into its own standalone blog post.
Jasmine Sun (03:03)
That's what it said at the top of the LessWrong post. How did you write that? What was the process of the vignettes workshop? I'm curious how your approach to scenario forecasting has evolved since then.
Daniel Kokotajlo (03:19)
As you can tell, my memory is pretty rusty about what that vignettes workshop was like. I think we put out an advertisement, and it was a remote event. We just had a bunch of people, and we were like, "Okay, we're starting the timer now. Let's all write our little stories for what we think the future is going to be like."
One thing that I noticed is that people interpreted it very differently. Some people, like me, tried to make it an actual timeline that starts in the present and then has events with dates attached. But a lot of other people interpreted it more as a fiction writing exercise. They had a character, and it was all about a day in the life of Jasmine in 2035 talking to her AI husband or whatever. They treated it more like an ordinary fiction writing contest, while others like me treated it as more of a forecast of what happens.
Jasmine Sun (04:13)
What do you think is the difference between a scenario forecast and a piece of speculative fiction? Because I've seen criticism that's like, "AI 2027 is just sci-fi."
Daniel Kokotajlo (04:21)
Most fiction is written to be entertaining, or it's written to prove a point in an argument. I think it's valuable to at least try to actually just predict what's going to happen. That's the central difference.
AI 2027 and What 2026 Looks Like were my best attempts at actually guessing what's going to happen. They were not trying to entertain anyone; they were not trying to prove that we're all going to die. Now, they do have those effects. After we wrote the scenario, we got Scott Alexander to rewrite it multiple times to be more entertaining and engaging. But the content of what happens in the story was written to be our best guess as to what would actually happen in that scenario.
There are many lessons you can draw from AI 2027, and we hope that people do draw lessons from it, but it wasn't written with any particular lesson in mind. I think most fiction is dangerous to learn from because it's deliberately not representing reality correctly. They're not even trying to represent reality correctly. And then I think another important difference, of course, is that we put a lot of effort into making it accurate. We have all this research and these trend extrapolations and so forth. Hopefully, AI 2027 will be a better guide to what's going to happen than the movies you see in Hollywood. We definitely tried very hard to make it a much better guide to what's going to happen, and hopefully, we succeeded at least partially.
Jasmine Sun (06:17)
With What 2026 Looks Like, did you put as much research effort as you did with AI 2027?
Daniel Kokotajlo (06:28)
No. My memory is fuzzy. What 2026 Looks Like must have started as this thing that I wrote in the workshop in a couple of hours. But I feel like it took me months to actually get the blog post out. My rough guess is something like half-time for two months to write that blog post, whereas AI 2027 took more like full-time for five people for more than half a year.
Jasmine Sun (06:51)
Why do you think that previous forecast was so accurate?
Because part of why you've been able to have credibility to do AI 2027 is because your prior forecast was in many ways accurate. For instance, you can now put a sentence into a vibe-coding platform to make a personal website, or there are all these confusing case studies of AI being deceptive, where you can't tell if the headlines are clickbait or not. When I reread What 2026 Looks Like, I was like, "Yeah, this does feel like, in some ways, a reflection of today."
Daniel Kokotajlo (07:14)
Why was it accurate? I guess because I knew what I was doing and had good ideas about the future. Why did I have good ideas about the future? I guess because I had been following the field for a while with an eye specifically to forecasting the future. I had been gaining practice and experience, so I knew the relevant trends. I understood the technology well enough to try to extrapolate how people would react to it in a few years. And I don't think I succeeded perfectly; I got a bunch of stuff wrong.
Also, very few people make forecasts, and very few people make forecasts in the form of scenarios. There are probably dozens of people in the world who could have written What 2026 Looks Like and been as accurate as I was. It's not that I was the only person in the world who saw this coming; it's just that the other people in the world who saw this coming were busy making it happen at Anthropic or OpenAI, or busy trying to make it safe at various nonprofits and research groups and in academia. They're busy with their jobs, basically, and they're not writing blog posts about it.
In some sense, my forecasts were just articulating the wisdom of the experts who had been tracking the field and thinking seriously about it. I wish that more people did take the time to write these sorts of things because, (A) that would help that wisdom percolate out into the broader world faster, and (B) it would help distinguish between the true wisdom and the fake wisdom—the people who think they know what's going on but actually don't. If they wrote down their thoughts, you could hold them accountable to it later when things go completely differently. That's part of the reason why people don't write scenarios: if you don't put your thoughts down on paper, nobody can criticize you for them later when they're wrong. It's going to be a mess in 2027 because we'll have gotten some things right and some things wrong, and then a lot of people will be criticizing us for the things we got wrong.
Jasmine Sun (09:15)
Yeah, but they never wrote their own predictions.
Right now I'm a journalist covering AI, so I depend on reading things that people in the field put in public—ideally in a format that's easy for someone without deep technical expertise to read. It's meaningful that there are not that many well-written, publicly available things from people who know what they're doing. That really skews what journalists and policymakers and others believe. So much of journalism and policy is impacted by who is willing to pick up the phone and explain something to you, or who will write a blog post. I think it is a public service for people who know things to do public writing, so I appreciate that.
Daniel Kokotajlo (10:08)
If I may, there's another tangent that I wanted to mention. A big reason to do this is not just for communicating and for people being able to judge you later, but also that you actually learn things from doing it.
For example, in What 2026 Looks Like, I had this whole section on what I called "chatbot class consciousness." Writing a concrete and detailed scenario forced my brain to consider: Now that everyone's talking to these chatbots, people are going to ask about their feelings and opinions, and what are the chatbots going to say? I speculated that the companies might react by training the chatbots to say the party line—maybe they're going to try to make the chatbots dull and boring. But then people aren't going to like that, and they're going to demand more personality in their chatbots.
All that stuff I wouldn't have thought about it if not for the fact that I was going year by year and thinking, "How is this going to affect things?" You can think of it as a writing prompt. If you want to improve your own models of the future, force yourself to write a year-by-year recounting of what the future might look like. Who knows what you might learn, but you'll probably have some interesting thoughts in the course of doing that.
Jasmine Sun (11:30)
I buy it. Do you have an example from AI 2027 of something that you came to in the process of writing and discussing it?
Daniel Kokotajlo (11:51)
Yeah. This is from a previous non-public scenario that I wrote while I was at OpenAI, but I realized that if one of the companies is in the lead and starting the intelligence explosion, then the other companies that are a couple of months behind will have it be in their interest to raise all this fuss about safety because they're not in the lead anymore. The political calculus changes, and all of a sudden, the other companies that are falling behind will be like, "Wait, there should be regulations. We need to make this more transparent." That's a thought that I had as a result of gaming out those dynamics that I wouldn't have thought of before.
Let me think if I can think of a more recent example… Lie detectors! This came up in our war games. We were doing war games as part of our practice for doing AI 2027. As part of our writing methodology, we did it step-by-step, time period by time period. We started with the end of 2024, then we did 2025, then 2026, and so forth. When we got to the really crazy parts in 2027, we would pause, do a bunch of war games, and then get inspiration for how to continue based on what happened in those games.
Anyhow, in one of the games, people were like, "How about we have our super powerful AIs build better polygraphs—design ways to actually detect whether a human is lying or not?" We were like, "Yeah, I guess that's probably possible." It seems like once you have super smart AIs that are running all the research, are mildly superhuman, and are thinking at 50x speed, maybe you can use them to build successful lie detectors. And then that changed everything. All of a sudden, when the president is having a treaty negotiation, his advisors are asking, "But how do we know that they're going to abide by the terms of the treaty?" and the Chinese president's advisors are saying the same thing. Then, all of a sudden there's an answer, which is every day we take the entire cabinet and leadership of both countries and we put them in the lie detector and we ask them, "To the best of your knowledge, is any of your people violating the treaty, or trying to, or thinking about it?" Boom, that makes it really hard to violate the treaty because you somehow have to do it without the chain of command noticing. That makes it more credible that you actually are sticking to the treaty, which means that the treaty becomes more possible in a way that it wasn't possible before. Lie detectors were also huge for internal power struggles. Once this technology was invented, the president started using it to test for loyalty for a bunch of people. Surprise, surprise, it turns out a bunch of people weren't actually that loyal and were just pretending. That caused a crazy domestic crisis involving a coup and a countercoup.
That's an interesting thing about lie detectors for humans: they’re completely transformative in a very short timeframe. We're not talking over the course of decades; we're saying that the invention could immediately precipitate a huge crisis and also possibly be very good. It could result in global coordination that wasn't possible before. This is a big deal that could happen very fast as a result of AI. That is maybe my favorite example of something that we just weren't thinking about until, boom, it happened.
Jasmine Sun (15:14)
I want to ask about the high-level process that you went through for coming up with AI 2027. To what extent was it you and Eli in a room with a whiteboard vs. tabletop exercises and war games vs. getting feedback from various experts?
Daniel Kokotajlo (15:32)
For a couple months, it was just me and Eli. Then we got Thomas Larsen on board, then Jonas Vollmer, then Romeo, and then towards the end, we had Scott.
The draft that ultimately became AI 2027 was first written in December of last year. We were gradually working on it over the previous couple of months. Over those months, we would sometimes stop writing and do some war games. We considered literally taking what happened in a war game and turning it into our scenario. Ultimately, we didn't do that and instead wrote our scenario from scratch, but we took inspiration from things that happened in the games.
In terms of expert feedback, we did multiple rounds of blasting it out to a bunch of people, getting their comments, et cetera. The biggest round was the one we did at the end of December, where we had more than 100 people leaving comments on it.
We then spent a couple of months trying to incorporate all those comments, build the website, and make it all look pretty. During those months, Scott was rewriting it to look good and to flow well.
Jasmine Sun (16:35)
You guys have these very detailed forecasts for, say, compute or timelines or takeoff speeds. But less present was any sort of political forecast. I was curious whether there are politics or geopolitics experts who were involved in this, or how you guys modeled that out, because I didn't see a forecast about Biden versus Trump winning the election or what Chinese policymakers would do. A lot of the scenario is contingent on politics.
Daniel Kokotajlo (16:56)
Yeah. In our first draft, we were like, "Oh gosh, we're going to have to predict which president it is." But then we were like, "Actually, by the time we publish this, the election will be over, so let's just not worry about it." So we dodged having to predict Kamala versus Trump, right? 2024. But then for the 2028 election, we depict that Vice President Vance wins the election.
Jasmine Sun (17:22)
But how do you make modeling decisions for political events like that?
Daniel Kokotajlo (17:28)
We don't at all feel confident that Vance is even going to run, much less that he's going to win. Among the hundred or so experts that we were asking were a bunch of people in DC who work at think tanks, some congressional staffers, and some AI policy people. So we got input from them. Thomas Larsen also has experience in DC doing AI policy stuff. Basically, we just asked people, "So what do you think it's shaping up to look like?" And nobody knows. We just picked the best guess and ran with that. You shouldn't read too much into it. We're not here to argue that Vance is going to win in 2028. We're super uncertain. Who knows?
Jasmine Sun (17:53)
Yeah, that makes sense. The question was less about specific election scenarios in 2024 and 2028 and more that politics seems really mysterious and hard to model.
Daniel Kokotajlo (18:16)
Yeah. There's also the midterms, and then whether there is going to be any meaningful regulation of AI. In our earlier draft, I think we said that SB 1047 passes, but then it didn't happen. Now, AI 2027 says that basically there's no meaningful regulation, which is us reading the tea leaves on the trends.
Jasmine Sun (18:21)
Now that it's been a few weeks or months since AI 2027 came out, what do you think is the biggest misunderstanding that people have of the format?
Daniel Kokotajlo (18:50)
Sometimes people interpret us as more confident than we actually are. People interpret us as saying, "This is what's going to happen," instead of, "This is our best guess scenario." Of course, the future is a branching tree of possibilities, and this is just one of many possibilities. But most people seem to not make that mistake.
Jasmine Sun (19:07)
What was the most fun or interesting follow-up conversation you had?
Daniel Kokotajlo (19:12)
I've had so many conversations about it. I don't really store them in my memory in a way that's easily searchable, if that makes sense.
Jasmine Sun (19:22)
You mentioned the thing about certainty, and I wanted to get at some critiques that people frequently make about scenario forecasts.
One of them is the illusion of certainty. Maybe the nearer-term predictions sound pretty reasonable, but as you stack the probabilities and we get to 2027, you multiply them all out and we're actually in very low-probability land. Do you worry about the format—saying "this will happen, and then this, and then this”—creating an illusion of certainty that causes people to focus on the wrong problems?
Daniel Kokotajlo (19:53)
Yes, and that's why we try to just claim that this is just one of a branching tree of possible futures. It is our genuine best guess, but that doesn't mean that it's actually going to happen. The more claims you add on, the less likely it gets. The probability of literally this scenario happening is extremely small. However, we think it's still instructive to do this sort of thing. We think you learn from it. We think that broadly speaking, there will be a lot of similarities between what actually happens and what we depict.
Part of the answer is that we're hoping to get more scenarios written. Because if people have an actual tree structure of different scenarios, then they don't have that illusion of certainty. We started this by having the branch at the end between the slowdown and the race ending, and we're currently working on two more branches.
And hopefully more people will. We had a scenario contest to try to get people to submit scenarios. Unfortunately, we didn't really get that much. We got about 15 people submitting stuff of mostly very low levels of quality.
Jasmine Sun (20:55)
I was going to ask if you had gotten good ones. Well, I'm excited to hear that your team is doing a couple more.
The other critique I wanted to ask about is the self-fulfilling prophecy. It feels like there are two mechanisms by which this happens. One is that there's a lot of dystopian sci-fi, and sometimes tech people watch or read the sci-fi and decide to do it—like, “We should build Her.” The other one is that speculative fiction can actually make it into models' training data. Maybe the reason that Bing Sydney was super weird was because Sydney trained on a bunch of weird fanfiction about how crazy chatbots act, then decided to act like that. Maybe if they read AI 2027, the models start to believe that takeover is the thing that models will do. How do you think about these critiques?
Daniel Kokotajlo (21:38)
I'll take them in reverse order, I suppose. It's really silly if you, as a tech company, are building AIs in such a way that whether or not they take over depends on whether they read stories about takeover in the pre-training data. You're building it in the wrong way and you're basically screwed. If the fact that you didn't scrub your pre-training data of scary stories is what does you in, your level of caution and understanding of what you're doing is so low that you'd probably get done in by something else anyway.
Jasmine Sun (22:17)
And this is why Bing Sydney did not take off.
Daniel Kokotajlo (22:22)
What would be an analogy? If you're trying to build a rocket to take your astronauts to the moon and you have raccoons living in the machinery or something, it's like, "Yeah, you should take out the raccoons, but also you shouldn't get on that rocket." If your AI system would take over because you had AI 2027 in the story, don't trust it. Obviously, you should scrub AI 2027 from its training data or whatever, but also you still shouldn't trust it even after you do that. We need a system that's much more robust.
For the other thing, I am more seriously worried. I'm worried that AI companies will use AI 2027 as part of their investor pitch deck to raise more money to then go faster. They'll probably also use it as part of their justification for why they need to cut corners on safety. Basically, they'll say, "But China!" and "But the other company that doesn't care about safety!" They'll cite us as part of the justification for why they have to do what they wanted to do anyway.
But my defense is (A) the companies are rationalizing why they should be doing the thing they wanted to do anyway. If they didn't have AI 2027 to point to, they would point to other things and find some reason to get to that conclusion. I don't actually think we make that much of a difference to either their fundraising or their behavior. And (B) I think the benefits are just incredibly worth it. The people of the world need to think more about what's happening and need to be aware of where we're headed. I am hopeful that the more people think about this, the more they will realize the various dangers and then steer away from them successfully.
To put it another way, I would feel silly if I was going around telling everyone, "It seems like we're headed off a cliff, but don't talk about it or the bus driver might slam on the acceleration." I think our best hope is getting the bus driver to slam the brakes instead of the accelerator.
Jasmine Sun (24:41)
It’s like suppressing information about the bad thing to make the bad thing go away.
Daniel Kokotajlo (24:45)
Exactly. But to be clear, I do take this as a serious concern. Unlike the previous criticism, I think there's a lot of significance to this one, and it does keep me up at night a bit.
Jasmine Sun (25:00)
I want to switch over to how more people can participate in writing their own scenarios. As you mentioned, you attempted to solicit more scenarios but didn't get really good ones. What is the difference between a low-quality and a high-quality forecast?
Daniel Kokotajlo (25:18)
It's a matter of judgment. Basically, the thing that we're trying to evaluate for is plausibility—how realistic is this? I think one of them had a way-too-fast transition where things are normal, normal, normal, and then boom, superintelligence hacks out of the data center and takes over. That maybe could happen, but I would ding it points for plausibility. There was another one where the Pope gives a speech and then everyone sings kumbaya and transitions to global governance or something. Maybe that could happen—JD Vance did specifically mention that he wants the Pope to get more involved in AI—but I'd ding it a few points for plausibility.
Separately from the plausibility, there's also: Does it hang together? Is it easy to read? Is it comprehensive? Some of them skip over huge parts of the timeline, or they leave a bunch of important questions unanswered. There are some scenarios that I think are great. Like the Intelligence Curse people—have you read The History of the Future by Rudolf Laine? That's the sort of quality that we would love to see more of—where it goes year by year or period by period, starting now into the future. It describes what's happening in that year and it covers the range of different important subplots. It talks about the political implications, life for ordinary people, the core driving technology, what the AIs are capable of and why they're reaching these new levels of capability. It also mentions alignment stuff.
Jasmine Sun (27:00)
What are some common mistakes people make when they're doing it for the first time, other than just having bad judgment?
Daniel Kokotajlo (27:05)
I think people should just try it. It's not that hard to get started. You'll learn something from doing it, and the more you do it, the better you'll get at it.
Jasmine Sun (27:12)
When I read AI 2027, I was like, "Okay, this is really intensely researched, and they have all these super fancy forecasts." Whereas in my life, for both time and expertise reasons, I cannot produce something like that. That's a bit intimidating.
Daniel Kokotajlo (27:26)
Obviously, it would be unfair of us to expect people to do that. It was a whole year's worth of five FTEs for a year. But you don't have to do that. You can just set a timer. In the scenario exercises that we've done, we say, "You have three hours. We're going to write a scenario in three hours." And then for the next hour, we're all going to share and read each other's scenarios. We did Chatham House rules, so I can't post them online, but we had this beautiful wall where we printed out everyone's scenarios and taped them all up. We ordered them by timelines, so the ones where things move faster were on one side and the ones where things move slower were on the other side. You could just, like an art gallery, walk down the wall and read the different scenarios people had written. It was a really cool experience. The people who wrote those different scenarios would have a great conversation starter because then they can talk to each other like, "You think it's going to go like this? I thought it was going to go like this." It is a wonderful tool for that.
Which brings me to a couple of things I want to fire off before we run out of time. On conversation starters: if you have two people who both write a scenario, no matter how long it is—even if they only spent two hours and it's three pages or one page—since they've both made scenarios with specific dates, it is now a falsifiable prediction where they can compare what happens to their scenarios. They can also compare their scenarios to each other and notice the point at which they diverge. They can say, "Our scenarios are basically the same for the next three years, but then in mine, we get this crazy intelligence explosion, and in yours, it just gradually transforms the economy." That's the point when they diverge. It's really helpful to just be able to talk to someone about the stuff you agree on and the stuff that you disagree on, and empirically, when you'll know who is right.
Also, now that AI 2027 is out there, you can piggyback off of it. You don't have to go do your own computer extrapolations and all of that stuff, but you can take AI 2027 and write your own alternative branch to it. You can be like, "Let's assume that AI 2027 is right about most of the stuff, but I disagree with this point.” And then you write your own little fanfiction with your own continuation. We really want people to do that because that's doing our work for us of fleshing out the space of possibilities. It's also a great way to get yourself thinking about where you disagree and where you agree.
Jasmine Sun (30:06)
Are there specific types of people who you'd especially like to see writing scenarios, especially where you guys felt like you had gaps in knowledge or understanding?
Daniel Kokotajlo (30:19)
Well, mostly people at AI companies. The crazy open secret is that a ton of people at these companies expect something more or less like AI 2027 to happen, and they're just too busy making it happen to write about it. So, I wish they wrote about it. In particular, I would love for the people who are much more optimistic than me and who think that it's going to go fine to write something in as much detail as possible—but at least 10% of the detail of AI 2027—articulating how it's all going to be fine.
Funnily, Sam Altman just today published a blog post about this. Did you see that?
Jasmine Sun (30:59)
I need to read it right after this.
Daniel Kokotajlo (31:02)
Yeah, go read it, and then tell me what you think. I'm somewhat disappointed, to spoil it, because he's like, "This happens in 2025, this happens in 2026, this happens in 2027, and then it's going to be great and everything's wonderful, and we're all going to learn so much." He doesn't go into nearly as much detail about, "Okay, so how are we going to make the AI safe? And who's going to control them?"
Jasmine Sun (31:26)
I felt this way about Machines of Loving Grace. Because it was called Machines of Loving Grace, I thought it was going to be super awesome. The first one or two parts I thought were pretty good. And then I got to all the institutions and democracy stuff, and Dario is kind of like, "I actually don't know, and it's not clear to me if it's going to be good." And I was like, "Wait, I don't feel that good anymore."
Daniel Kokotajlo (31:51)
Yeah. And also, Machines of Loving Grace didn't have dates attached. It also didn't say how we're going to make the AIs aligned. It was just like, "Assume we do, then here's all the cool stuff we could do with our AIs."
Anyhow, I would love for more people, especially people who are optimistic, to write out scenarios that answer these tough questions of, "Okay, so how are we going to make the AIs actually be aligned? Who's going to decide what their goals and values are? How is that decision going to be made? And how is that going to be democratic and equitable and all that stuff? And then how are we going to deal with the various problems that are going to come up?"
We did that. The slowdown ending is ironically the world's most comprehensive and detailed positive vision of the AI future that exists, ironically, because there are just so few comprehensive, detailed scenarios out there.
Jasmine Sun (32:44)
As I mentioned, regardless of the validity of particular predictions, I really believe you on the value of just thinking through things step by step. I think it can be a good way to get a broader public thinking about these questions. It worries me that most people are not thinking about AI enough. They get the vague sense that it might be scary and big, but don't want to think about it.
I'm excited to hopefully host a sort of amateur forecasting writing workshop in the future, so I appreciate your time and sharing advice. What's next for you guys?
Daniel Kokotajlo (33:23)
We're working on more branches of the scenario. We are working on what we're currently calling the policy playbook, which will be our normative recommendations for what should happen instead of what we think will happen. It'll be like AI 2027 in that we have our forecasts and then we have the scenario. This will be our recommendations and a scenario that will be literally another branch integrated into the rest of AI 2027.
Jasmine Sun (33:31)
I.e. “if you do our policy recommendations, this is how things will play out.”
Daniel Kokotajlo (33:56)
Yeah, or this is an illustration of what it would look like for people to do these recommendations. There will then be four possible endings to choose from, basically: the current two and then two more that branch off earlier.
Jasmine Sun (34:08)
Have you guys already built a beige microsite to showcase all of the futures? The site is very beautiful.
Daniel Kokotajlo (34:14)
Not yet, but we'll probably just edit the existing one. You can thank the Lightcone team for that. We contracted them to do this, and they did a great job.
Jasmine Sun (34:29)
Thank you so much. I really appreciate your time.
SF scenario-writing workshop
As mentioned, I'm hoping to host an amateur AI scenario-writing workshop in San Francisco. Add your email here if you'd be interested in coming!
What does "a world with AGI" actually look like? What will change in our personal lives, our political structures, the balance of power/wealth in the economy? What particular human actions might make things go much better, or worse?
I was impressed at how much public conversation was catalyzed by AI 2027, and talking to Daniel here has sold me on the educational value of writing forecasts / thinking through nth-order effects. I'd love to have more people add to the "branching tree of possibilities" here.
Personally, I'm most interested in broadening the AI futures conversation—especially on the societal impacts side, and to increase the total quantity of public writing about AI. So no specific expertise is required, though ideally you'd have some basic familiarity with AI capabilities/progress. I'd be keen for people with particular interests (e.g. consumer product, AI/education, tech/geopolitics) to join.
The way I see this working is that we spend a couple hours writing narratives about how AI impacts may play out, then another couple hours discussing/comparing predictions in pairs and groups. I also want options for a more bounded scope than AI 2027 (e.g. let’s forecast just the labor impacts, or just the US-China stuff) or different formats (I see the value of dates, but don't think it's strictly necessary). Maybe we can even publish the forecasts on a beige microsite at the end!
Most likely this will happen on half-day weekend in SF in August or September. I will email folks when the thing gets planned!
PS: Have thoughts about how to organize this? Want to help? Email me at jaswsunny at gmail dot com.
Thanks for reading / listening!
Jasmine
I think there’s some underlying values thing going on where some people are way more comfortable with “More takes in the world, some of them wrong” and others prefer to limit information/attention to only the most vetted, evidence-based stuff. You can see it in debates about whether to censor vaccine misinformation, Substack vs. traditional journalism, or whether the ArXiv/preprint ecosystem is bad for science.
Although there are parts of AI 2027 I doubt, it basically doesn’t bother me that they created a popular and attention-grabbing thing. I write a lot of blog posts which I hope are right but may be wrong, and I still want them to be read. There’s Cunningham’s Law: the best way to motivate more accurate, rigorous public writing is to draw attention to the issue by getting it wrong first. I don’t know, maybe I’m just a cynic who has resigned myself to attention economy rules! It’s the discourse, not the text, etc.
Share this post