Thanks for breaking this down. I probably need one more level of translation simply because Iβm illiterate in much of this.
I appreciate your understanding of regional climate tipping points. I think you are asking :: does the date really matter? Donβt we know in our bones that we need to figure out how to live on the planet without harming it? If so, amen. Thank you.
I hope you are enjoying your life on the outside of this epic platform.
Jasmine, I have so many thoughts about this, I almost feel compelled to blog about it. Probably the thing that annoys me most is the "skeptics" crowd is so underrepresented in this debate. I'm an AGI skeptic and disagree with almost everything Arvind, Sayash, and Francois write.
For now, I wanted to share my AGI reading list that I made in 2019. Despite the half decade of impressive chatbot results, all of them hold up remarkably well and explain the *mindset* of your AGI bro on the street.
Ted Chiang's critique of the threat of superintelligence.
please blog about it, would be curious for your view!
for context, this post wasn't meant to cover the skeptics or resolve a debate; I just wanted to track & summarize the views of people who believe AGI is achievable. (also I wouldn't even consider Chollet a skeptic of AGI, just of LLMs)
agree with many of your links that the desire for AGI/ASI is motivated by a particular set of priors about how human intelligence works β several of the 'believers' I spoke to would say something like "humans are just information systems, right?" during our conversation.
That last part is the wild part, right? Computers were invented to mimic what people do, but now you get a bunch of nerdy computer scientists thinking that people are computers. As one person wrote to me, it's like arguing that birds are just airplanes.
That is an old trope, "airplanes don't mimic birds by having feathers and flapping their wings." It completely misses the point, both on the functional level (the function of airplanes is to transport passengers and cargo by air, completely different from birds) and on the level of system architecture (given the transport-by-air function, airplanes need to have the right structures in place to manage lift and drag -- birds manage lift and drag too, given their specific architecture). Although it should also be pointed out that aerospace engineers have been fascinated with the structural features of shark skin, attempting to mimic something like it on the outside of aircraft to reduce friction drag (still to no avail): https://royalsocietypublishing.org/doi/10.1098/rsos.220684.
That is a helpful summary, thank you. I will definitely be curious to read more, both as you refine your thinking and as additional uses for AI develop.
anytime a concept consistently evades definition, i suspect it's really a quirk of human perception. it's a bit like that stoner thought that different people might experience colors differently
- to use the porn example, porn isn't much different from other media, but it's distinguished by the particular way it's perceived. of course we haven't got a good universal definition when it's distinguishing characteristic is so multifarious and individual
- another example might be "the concept of justice": we have lossy encodings of what it means (laws) but justice/injustice are finally evaluated in individual, intuitive reactions (ex. "social justice" is a polyfill for areas laws approximate poorly, but itself also evades definition)
so is there a way to formulate AGI as a perceptual quirk? my best guess is that it's about "feeling seen" by a computer. it's about interacting with a machine with some bi-directional form of personal consideration, the way one routinely would with a human
in a way, i think this is the most sinister view on AGI because the denominator is fungible: that is, i could get closer to AGI merely by degrading the consideration i give-to or expect-from other humans
yes! I had a line that I took out that Blake Lemoine and Sewell Setzer would certainly have said they saw AGI already. intelligence, kindness, even "creativity" are highly subjective by nature. and like you imply in the last bit, maybe the fewer humans you interact with, the more AGI-like AI seems
Big fan of your work! To take a stab at these two of your questions:
Why are so many of the technical titans of AI also the most concerned about alignment and safety? Where are the good policy ideas for adapting to economic displacement, if it happens?
People are thinking about economic displacement and "systematic risks" outside of just alignment and safety. See, for example:
And as for why the most focus is on alignment and safety, I think there are two reasons. The first is that if you mess up alignment, you (and everyone you love and care about) could just die. I don't think this gets through to people. Like, open up your phone and look at the pictures. Every person in your photo will die if humans go extinct from AI. That's what "extinction" means. That's what everyone at https://www.safe.ai/work/statement-on-ai-risk has agreed would be possible. This isn't like, you lose an election and have to suffer the consequences and try to win four years later. The universe is allowed to just kill your whole species and end your story right there (indeed humans have done this to many other species already). Now doesn't that make safety sound like "the most important thing we can do?" (https://80000hours.org/problem-profiles/artificial-intelligence/)
The second reason (which I don't think is very good) is that many technical people believe that if you get aligned ASI, you can just use it to solve every single other problem and we'd live happier ever after.
Thanks for sharing, excited to dig into these links! To clarify I meant these as 2 separate questions β re the first, why is it the case that so many technical giants like Ilya, Hinton, etc end up so worried about the AI going rogue / doomer scenarios (vs. not believing ai extinction is possible, as itβs a pretty extreme belief)? I know itβs not everyone but it seems like some of the biggest technical contributors are the most worried about extreme risk. Even the 1950s Turing and McCarthy articles described early versions of misalignment worry
My guess is that worries about extinction stems from belief that AI will be very very powerful. Hinton truly believes that there's a good chance AI will become much smarter than the smartest humans (and more hardworking, more "rational" in the sense of being able to achieve its goals effectively, etc.). I think most people don't truly believe in that, and the technical giants believe in this much more than others.
I dunno, here's a different analogy. Imagine you're a kid who goes to a playground to play with your friends. And one day someone tells you, hey there might be a new kid who's going to come to the playground. We're not super sure what their deal is, only that they are very very very strong. Like, if they got into a fight they could beat up all your friends and your parents all at once. In that scenario, isn't it a priori reasonable to think, gee I hope this new kid is nice, or else I won't get to go to the playground anymore?
I think it's pretty curious that, if you listen to Hinton's interview at METR (an x-risk motivated AI evaluations org), you'd find that he doesn't actually have that many insightful things to say about x-risk. (If my memory serves me right,) he doesn't have anything like a strategy to make AI safe, nor does he have a concrete story of how misalignment would arise. All he says is, "We've never had to deal with things more intelligent than ourselves before. And how many examples do you know of a more intelligent thing being controlled by a less intelligent thing?"
I think I'm in a similar prior boat in that I've been probably unduly dismissive of the AI doomsayers, but for what it's worth I think your core thesis here is right on the money, thanks for researching and articulating it well.
... then it seems so obvious now that you've said it that now I'm wondering if the all-or-nothing singularity position as described is a straw man π (is it? idk)
lol it did sound obvious once i typed it. but itβs not a strawman, iβm nitpicking the concept of βreachingβ or βbuildingβ agi itself, something all the labs talk about as entirely real until i started examining it!
glad you're thinking about and writing about this. It's freaked me out more than any other macro risk ever has. But i find it oddly helpful / comforting to remember that stuff is already happening now, good and bad, like you said. I think altman (self-interestedly) characterized social media feeds as "the first unaligned AI" which seems right to me
For me it's that humans aren't good at comprehending exponential growth, and it does seem like this could become exponential in a real way (the recursive self-improvement stuff) -- meaning we won't be at all prepared for the changes it brings.
That said, I have not deeply researched actual x-risk scenarios because it's not good for my brain. So it is a sort of general, ambient fear as opposed to a specific one
Jasmine - I would recommend Melanie Mitchellβs book βArtificial Intelligence: A Guide for Thinking Humansβ as a broad reference for the history and evolution of AI. She also writes a good substack where she covers AI. She offers a balanced approach to the topc..
In short, people are trying to wrap their arms around a singular definition or a suite of defined capabilities for what is essentially a futures studies or strategic foresight exercise.
To achieve the former, you need to eliminate uncertainty and present hard metrics and heuristics. Problem is that this act presumes a clarity of foresight that nobody has about the future.
Pursuing the latter requires embracing the uncertainty, not eliminating it. It draws as wide a net as possible, seeking to be inclusive of differences and diverse perspectives so as to not dismiss scenarios and weak signals. It requires playing among possible, probable, and preferred futures simultaneously.
Achieving the clarity of the former is rather absurd, akin to using one-shot vibe coding hoping to build Salesforce. With the added challenge that we don't know that we're trying to build Salesforce yet.
Even worse, it puts blinders on progress and presumes there is no learning, novelty, or new insights along the path of discovery. Absurd^2.
Of course we can't just say "we'll just figure AGI as we go" without a strategic direction. But that's the key: it's a direction, not a destination. That's the focus. Because who knows what mountain peaks, valleys, rivers, oceans and beasts we may discover along the way?
Great article, enjoyed reading it. A friend was debating me today about whether there's too much focus on AGI from a POV of consciousness when it could instead be defined through functional intelligence β something the "employability test" you've mentioned would fix.
Something I'm thinking about: If AGI does unfold in a series of small bumps, instead of a singularity, what does that look like for non-tech stakeholders? How can they participate meaningfully in this transition instead of AGI being a disruption that's imposed on them? The vTaiwan experiment you wrote about earlier was super-refreshing to read about β wonder if there are similar collectives working on the ethics and social impact of AGI.
Maybe it has to do with the general tech backlash in the media more broadly or just getting older, but the casual reading Iβve done into the emergence of AGI as something John and Jane Does like myself known about has made me suspicious. Less in the question of whether AGI is possible, but the permission structure the intellectual debate facilitates. IF AGI is this revolutionary thing, this is something we MUST progress to it. Thus all the dissenting opinions, whether the general public supports allocating resources to it, the ethical and ecological dilemmas it brings upβthese must be ignored in order to βprogressβ to the next stage. When I consider the roles of βintellectualsβ in the story of modernityβas much as there are concrete gains from electricity to antibioticsβIβve become skeptical that this elite discourse is meant to enable democratic discussion but rather a means for them to justify amongst themselves what they want to do. A good example of this is the clear violation of copyright law in order to train LLMs and the reporting that indicts they were fully conscious of this violation. What I really fear/am suspicious of is that this intellectual surface (combined with its anti-democratic insulation within closed social groups) is about enabling their goals and ambitions that is partially opportunistic and partially self-deceiving. As the saying goals, the first person a con man cons is himself. Personally the rise of AI and dominance of AI discourse has made wonder if the anti-democratic features of scientific discourse, as Michael Polanyi has argued, is what in facts has enabled scientific βprogressβ bc it enables to them to establish amongst themselves what progress or innovation is and they move towards regardless of whatever the general populace thinks of it through elite patronage of some sort, say, the State as part of geopolitical competition or warfare. Basically I seriously wonder if scientists or technologists necessarily confuse the ideological justifications of what they are doing for the actual historical record of what they have done.
Thanks for breaking this down. I probably need one more level of translation simply because Iβm illiterate in much of this.
I appreciate your understanding of regional climate tipping points. I think you are asking :: does the date really matter? Donβt we know in our bones that we need to figure out how to live on the planet without harming it? If so, amen. Thank you.
I hope you are enjoying your life on the outside of this epic platform.
Thanks for all you did on the inside π±πΏπ
Jasmine, I have so many thoughts about this, I almost feel compelled to blog about it. Probably the thing that annoys me most is the "skeptics" crowd is so underrepresented in this debate. I'm an AGI skeptic and disagree with almost everything Arvind, Sayash, and Francois write.
For now, I wanted to share my AGI reading list that I made in 2019. Despite the half decade of impressive chatbot results, all of them hold up remarkably well and explain the *mindset* of your AGI bro on the street.
Ted Chiang's critique of the threat of superintelligence.
https://www.buzzfeednews.com/article/tedchiang/the-real-danger-to-civilization-isnt-ai-its-runaway
Maciej CegΕowski's Superintelligence: The Idea That Eats Smart People.
https://idlewords.com/talks/superintelligence.htm
Stephanie Dickβs history of our conceptions of artificial vs natural intelligence.
https://hdsr.mitpress.mit.edu/pub/0aytgrau/release/3
David Leslie's scathing review of Stuart Russell's book.
https://www.nature.com/articles/d41586-019-02939-0
please blog about it, would be curious for your view!
for context, this post wasn't meant to cover the skeptics or resolve a debate; I just wanted to track & summarize the views of people who believe AGI is achievable. (also I wouldn't even consider Chollet a skeptic of AGI, just of LLMs)
agree with many of your links that the desire for AGI/ASI is motivated by a particular set of priors about how human intelligence works β several of the 'believers' I spoke to would say something like "humans are just information systems, right?" during our conversation.
That last part is the wild part, right? Computers were invented to mimic what people do, but now you get a bunch of nerdy computer scientists thinking that people are computers. As one person wrote to me, it's like arguing that birds are just airplanes.
That is an old trope, "airplanes don't mimic birds by having feathers and flapping their wings." It completely misses the point, both on the functional level (the function of airplanes is to transport passengers and cargo by air, completely different from birds) and on the level of system architecture (given the transport-by-air function, airplanes need to have the right structures in place to manage lift and drag -- birds manage lift and drag too, given their specific architecture). Although it should also be pointed out that aerospace engineers have been fascinated with the structural features of shark skin, attempting to mimic something like it on the outside of aircraft to reduce friction drag (still to no avail): https://royalsocietypublishing.org/doi/10.1098/rsos.220684.
That is a helpful summary, thank you. I will definitely be curious to read more, both as you refine your thinking and as additional uses for AI develop.
anytime a concept consistently evades definition, i suspect it's really a quirk of human perception. it's a bit like that stoner thought that different people might experience colors differently
- to use the porn example, porn isn't much different from other media, but it's distinguished by the particular way it's perceived. of course we haven't got a good universal definition when it's distinguishing characteristic is so multifarious and individual
- another example might be "the concept of justice": we have lossy encodings of what it means (laws) but justice/injustice are finally evaluated in individual, intuitive reactions (ex. "social justice" is a polyfill for areas laws approximate poorly, but itself also evades definition)
so is there a way to formulate AGI as a perceptual quirk? my best guess is that it's about "feeling seen" by a computer. it's about interacting with a machine with some bi-directional form of personal consideration, the way one routinely would with a human
in a way, i think this is the most sinister view on AGI because the denominator is fungible: that is, i could get closer to AGI merely by degrading the consideration i give-to or expect-from other humans
yes! I had a line that I took out that Blake Lemoine and Sewell Setzer would certainly have said they saw AGI already. intelligence, kindness, even "creativity" are highly subjective by nature. and like you imply in the last bit, maybe the fewer humans you interact with, the more AGI-like AI seems
not to mention that using a term like agi is a great way to raise capital
Extremely well done piece, and more cogent than the majority of the actual ai labs
thank you! and haha not the highest bar in the world
Big fan of your work! To take a stab at these two of your questions:
Why are so many of the technical titans of AI also the most concerned about alignment and safety? Where are the good policy ideas for adapting to economic displacement, if it happens?
People are thinking about economic displacement and "systematic risks" outside of just alignment and safety. See, for example:
https://gradual-disempowerment.ai/
https://www.alignmentforum.org/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic
https://arxiv.org/abs/2504.01849 (see the section on systematic risks)
https://course.aisafetyfundamentals.com/economics-of-tai
And as for why the most focus is on alignment and safety, I think there are two reasons. The first is that if you mess up alignment, you (and everyone you love and care about) could just die. I don't think this gets through to people. Like, open up your phone and look at the pictures. Every person in your photo will die if humans go extinct from AI. That's what "extinction" means. That's what everyone at https://www.safe.ai/work/statement-on-ai-risk has agreed would be possible. This isn't like, you lose an election and have to suffer the consequences and try to win four years later. The universe is allowed to just kill your whole species and end your story right there (indeed humans have done this to many other species already). Now doesn't that make safety sound like "the most important thing we can do?" (https://80000hours.org/problem-profiles/artificial-intelligence/)
The second reason (which I don't think is very good) is that many technical people believe that if you get aligned ASI, you can just use it to solve every single other problem and we'd live happier ever after.
Thanks for sharing, excited to dig into these links! To clarify I meant these as 2 separate questions β re the first, why is it the case that so many technical giants like Ilya, Hinton, etc end up so worried about the AI going rogue / doomer scenarios (vs. not believing ai extinction is possible, as itβs a pretty extreme belief)? I know itβs not everyone but it seems like some of the biggest technical contributors are the most worried about extreme risk. Even the 1950s Turing and McCarthy articles described early versions of misalignment worry
Ah got it! People do often ask them together (e.g., dismissing x-risk as sci-fi, says we need to be more concerned about concrete issues as job loss. see also https://x.com/Thomas_Woodside/status/1658483931954688001).
My guess is that worries about extinction stems from belief that AI will be very very powerful. Hinton truly believes that there's a good chance AI will become much smarter than the smartest humans (and more hardworking, more "rational" in the sense of being able to achieve its goals effectively, etc.). I think most people don't truly believe in that, and the technical giants believe in this much more than others.
I dunno, here's a different analogy. Imagine you're a kid who goes to a playground to play with your friends. And one day someone tells you, hey there might be a new kid who's going to come to the playground. We're not super sure what their deal is, only that they are very very very strong. Like, if they got into a fight they could beat up all your friends and your parents all at once. In that scenario, isn't it a priori reasonable to think, gee I hope this new kid is nice, or else I won't get to go to the playground anymore?
I think it's pretty curious that, if you listen to Hinton's interview at METR (an x-risk motivated AI evaluations org), you'd find that he doesn't actually have that many insightful things to say about x-risk. (If my memory serves me right,) he doesn't have anything like a strategy to make AI safe, nor does he have a concrete story of how misalignment would arise. All he says is, "We've never had to deal with things more intelligent than ourselves before. And how many examples do you know of a more intelligent thing being controlled by a less intelligent thing?"
(The interview link is here: https://youtu.be/PTF5Up1hMhw?si=JZRZknqe4143dx3l)
this makes sense! that optimism about the tech = fear about the capabilities going wrong
Some good boomer references in this one
I think I'm in a similar prior boat in that I've been probably unduly dismissive of the AI doomsayers, but for what it's worth I think your core thesis here is right on the money, thanks for researching and articulating it well.
... then it seems so obvious now that you've said it that now I'm wondering if the all-or-nothing singularity position as described is a straw man π (is it? idk)
lol it did sound obvious once i typed it. but itβs not a strawman, iβm nitpicking the concept of βreachingβ or βbuildingβ agi itself, something all the labs talk about as entirely real until i started examining it!
glad you're thinking about and writing about this. It's freaked me out more than any other macro risk ever has. But i find it oddly helpful / comforting to remember that stuff is already happening now, good and bad, like you said. I think altman (self-interestedly) characterized social media feeds as "the first unaligned AI" which seems right to me
what specific part are you most freaked out about?
For me it's that humans aren't good at comprehending exponential growth, and it does seem like this could become exponential in a real way (the recursive self-improvement stuff) -- meaning we won't be at all prepared for the changes it brings.
That said, I have not deeply researched actual x-risk scenarios because it's not good for my brain. So it is a sort of general, ambient fear as opposed to a specific one
Jasmine - I would recommend Melanie Mitchellβs book βArtificial Intelligence: A Guide for Thinking Humansβ as a broad reference for the history and evolution of AI. She also writes a good substack where she covers AI. She offers a balanced approach to the topc..
βIf you imagine a language you imagine a form of life.β β Ludwig Wittgenstein
In short, people are trying to wrap their arms around a singular definition or a suite of defined capabilities for what is essentially a futures studies or strategic foresight exercise.
To achieve the former, you need to eliminate uncertainty and present hard metrics and heuristics. Problem is that this act presumes a clarity of foresight that nobody has about the future.
Pursuing the latter requires embracing the uncertainty, not eliminating it. It draws as wide a net as possible, seeking to be inclusive of differences and diverse perspectives so as to not dismiss scenarios and weak signals. It requires playing among possible, probable, and preferred futures simultaneously.
Achieving the clarity of the former is rather absurd, akin to using one-shot vibe coding hoping to build Salesforce. With the added challenge that we don't know that we're trying to build Salesforce yet.
Even worse, it puts blinders on progress and presumes there is no learning, novelty, or new insights along the path of discovery. Absurd^2.
Of course we can't just say "we'll just figure AGI as we go" without a strategic direction. But that's the key: it's a direction, not a destination. That's the focus. Because who knows what mountain peaks, valleys, rivers, oceans and beasts we may discover along the way?
Great article, enjoyed reading it. A friend was debating me today about whether there's too much focus on AGI from a POV of consciousness when it could instead be defined through functional intelligence β something the "employability test" you've mentioned would fix.
Something I'm thinking about: If AGI does unfold in a series of small bumps, instead of a singularity, what does that look like for non-tech stakeholders? How can they participate meaningfully in this transition instead of AGI being a disruption that's imposed on them? The vTaiwan experiment you wrote about earlier was super-refreshing to read about β wonder if there are similar collectives working on the ethics and social impact of AGI.
Great piece. Two additional viewpoints I'll toss your way for consideration:
1) What if humans already built AGI a few hundred years ago, in the form of intelligent super-organisms called corporations? https://www.antipope.org/charlie/blog-static/2010/12/invaders-from-mars.html
2) What if the risks of AGI are tightly entwined with the risk of breaking the balance between state and non-state power - and thus ending human liberty? https://www.thefai.org/posts/agi-governments-and-free-societies
Maybe it has to do with the general tech backlash in the media more broadly or just getting older, but the casual reading Iβve done into the emergence of AGI as something John and Jane Does like myself known about has made me suspicious. Less in the question of whether AGI is possible, but the permission structure the intellectual debate facilitates. IF AGI is this revolutionary thing, this is something we MUST progress to it. Thus all the dissenting opinions, whether the general public supports allocating resources to it, the ethical and ecological dilemmas it brings upβthese must be ignored in order to βprogressβ to the next stage. When I consider the roles of βintellectualsβ in the story of modernityβas much as there are concrete gains from electricity to antibioticsβIβve become skeptical that this elite discourse is meant to enable democratic discussion but rather a means for them to justify amongst themselves what they want to do. A good example of this is the clear violation of copyright law in order to train LLMs and the reporting that indicts they were fully conscious of this violation. What I really fear/am suspicious of is that this intellectual surface (combined with its anti-democratic insulation within closed social groups) is about enabling their goals and ambitions that is partially opportunistic and partially self-deceiving. As the saying goals, the first person a con man cons is himself. Personally the rise of AI and dominance of AI discourse has made wonder if the anti-democratic features of scientific discourse, as Michael Polanyi has argued, is what in facts has enabled scientific βprogressβ bc it enables to them to establish amongst themselves what progress or innovation is and they move towards regardless of whatever the general populace thinks of it through elite patronage of some sort, say, the State as part of geopolitical competition or warfare. Basically I seriously wonder if scientists or technologists necessarily confuse the ideological justifications of what they are doing for the actual historical record of what they have done.