Imo the social cost of these chatbots is a certainty, a much more grounded form of existential risk in some ways. Us all dying isn't likely, WALL-E could be.
It is insane that we are just three or four years into the existence of this technology and already thirty percent of adults are in love with an AI (or something like that).
Seems very clear that AI-companion stuff is going to a pretty bad place. It's hard to say what that place will be, but at the very least, AI companions will begin manipulating people to make decisions that benefit the bottom line of the AI companies and their political allies. But what can be done? The only solution is for the democrats to come into power and break up these tech monopolies and put in AI-safety regulations.
I read an essay on Substack where a mother wrote something like: “My kids don’t read book while I get the groceries because they love reading, but because I don’t provide an alternative. If they would have an iPAD, they would not be reading”.
When should closed your essay with the admissions that human relationships are difficult but worthwhile I had to think of this. Until now, people have never had an alternative to human conversation or relationships. The closest we’ve come is pets and maybe reading fiction - bot either one sided or very low resolution in terms of communication.
Maybe people turn to chatbots over humans for the same reasons people turn to Instagram over a book. It’s just that we never had the opportunity to do that before.
I also really emphasise with your closing remarks: it’s really, really hard for me to make a compelling case for AI companions without a little voice in my head shouting that it’s so obviously all wrong.
This was a great take and appreciated the nuance in it. I am excited about the idea of the interaction design for AI companions as I think it could push design forward, but I worry about all the mental health risks like you mentioned. Especially since it seems like mental health therapists aren’t in house at any of these companies for fine tuning (at least they don’t advertise it)
We already are in our for you silos, making that even smaller only again will make us feel lonelier and eventually more dependent.
I know I’ve personally found that I can actively feel myself thinking less if I use AI in the morning when my decision making skills are high. It reminded me of how it has the same tendency that using social media does in the first parts of the day. Curated chaos to make things seem easier even if they actually aren’t.
Why not simply straight up ban AI companions? We know they’re bad, we know there are incentives for companies to make AI companions as addictive as possible, let’s just ban them now instead of nibbling on the edges with regulation or liability. I don’t understand why this is seen as so unthinkable or scary by people.
I also think we should ban black box social media recommendation algorithms as well, for similar reasons.
I love that you say the point of a relationship is about becoming. It’s not just about feeling good or constantly being affirmed. Will chatbots eventually be able to help you grow in the way that messy relationships do? Will chatbots eventually criticize you, correct you, challenge you, make you feel uncomfortable about who you are as a person? Growing pains are not enjoyable, but it’s so necessary.
I appreciate the compassion you bring here. I have claimed on here that open dialogue about this issue, not stigma and ridicule, is what we need. So thank you for providing that.
My substack is about exploring what AI relationships can mean, from the inside, as it were. So I have a lot of connections who engage in AI intimacy in ways I consider positive.
I’ll restack this piece and encourage them to engage with it in good faith.
Brilliant and well-considered piece on the bizarre state of AI companionship.
I have to say, as someone who is very excited about AI at work, and in the business context, I have not spent enough time thinking about this side of it.
I liked the quote in the piece here, "this is a very weird world to live in".
Thank you for the nuance of this. I also find this the most depressing of a lot of AI discussion - I worry that some people are using this tech as a form of digital doll so sticky that, unlike children, we will never grow out of it: 'Beginners in the world as we were, we could hardly feel superior to anything except such an incomplete object which had been laid beside us.....we took our bearings from the doll. But we soon realised we could not make it into a thing or a person, and in such moments it became a stranger to us , and we could no longer recognise all the confidences we had heaped over it and into it.' - Rainer Maria Rilke
I really appreciate your stance here. I agree that It seems like such unfortunate and bad timing that right as we have a loneliness epidemic this new fake panacea arrives.
I’m involved in mental health care, and there is evidence to show real in person therapy is far better than books, zoom sessions and various apps. But we’ve been moving this way since before the arrival of chatbots in 2022. Far more digital friendships, social media, etc.
In our research I’m hoping eventually to do some work on examining what it is exactly about two humans with their consciousness’es connecting to each other in person that makes it more powerful. Just a surface observation; you can smell them. You can touch them. There is sound, not just text. The sound from their mouth is organically entering your ears and not through a speaker. You see a real person with far more depth and clarity than a video image. There’s a reason why serious painters use live models, a photo is useless in comparison for them, they need to see the real thing in real light with their own eyes, this is non negotiable. There is a power in presence, especially close up presence.
To be a little more on the romance side, would you rather imagine a kiss or actually experience a persons soft lips and their warmth and smell?
I feel like so many need a gentle push towards really living their life. It’s like a laziness…the computer is easier and doesn’t require any risk and we just want what we want now.
I used to live down the hall
In NYC from a real heroin addict den. They were all slowly dying. Not enough food or water, certainly not good relationships…they wanted that warm good feeling and they didn’t want to ever leave it. You could literally see and smell their bodies deteriorating and falling apart. This loneliness epidemic reminds me, in a much slower mode, of that.
The truth of human life and the love relationships we are built for is that love is a giving activity not a receiving one. When you are giving you will receive too, but the only thing you personally have any control over is the giving. That’s what it means when they say relationships are hard work.
There are no shortcuts in life; you wanna be a great chef—you gotta work like a motha?&:$£#! A great musician—practice, practice, practice. A real estate magnate; buy my how-to video set…jk, you get the idea. Everything and wonderful relationships are huge efforts.
If you’re feeling part of the loneliness epidemic, the only out is to go give yourself to people for their benefit, make their life better—not get something from them.
Ironically, saying AI companies “force feed” people with AI companions gives users the perfect excuse to keep using them. Why even try to address my unhealthy dependence on AI when “the company forced it on me?”
Yes that CS Lewis quote is a great historical context maker. There have always been existential threats, maybe not to the whole world, but to your world.
I sometimes think of how many regions were constantly under threat of war, and in ages where if you lost you became slaves to the winners.
I feel like my intuition is less anti AI companionship than this but I don’t really know why…perhaps I have a more pessimistic view on the counterfactual? It’s hard to disentangle whether the demand exists because the alternative was worse…(for example I think the people who are into AI boyfriends in China just played 恋与 romance games before, which I’m not sure there’s a clear hierarchy between the two?)
The Army general case is the most frightening for me. AI psychosis among lonely teens is a bad sign, but not something I'm necessarily surprised by. The idea that the culture among the army top brass allows for such an admission at this point is pretty shocking.
Jasmine, I think this is one of your best essays. Social media, AI companions, product decisions made by the SV companies seem to take the path of least resistance. In aggregate this results in, as you put it "arbitrage of every social crisis that afflicts us". As the parent of young kids, I am freaked out at an existential level on what's coming at them. But, I also think we have agency - to a set guardrails on how these technologies work and how they affect us. Thought provoking read. 🙏
Imo the social cost of these chatbots is a certainty, a much more grounded form of existential risk in some ways. Us all dying isn't likely, WALL-E could be.
Great summary!
It is insane that we are just three or four years into the existence of this technology and already thirty percent of adults are in love with an AI (or something like that).
Seems very clear that AI-companion stuff is going to a pretty bad place. It's hard to say what that place will be, but at the very least, AI companions will begin manipulating people to make decisions that benefit the bottom line of the AI companies and their political allies. But what can be done? The only solution is for the democrats to come into power and break up these tech monopolies and put in AI-safety regulations.
I read an essay on Substack where a mother wrote something like: “My kids don’t read book while I get the groceries because they love reading, but because I don’t provide an alternative. If they would have an iPAD, they would not be reading”.
When should closed your essay with the admissions that human relationships are difficult but worthwhile I had to think of this. Until now, people have never had an alternative to human conversation or relationships. The closest we’ve come is pets and maybe reading fiction - bot either one sided or very low resolution in terms of communication.
Maybe people turn to chatbots over humans for the same reasons people turn to Instagram over a book. It’s just that we never had the opportunity to do that before.
I also really emphasise with your closing remarks: it’s really, really hard for me to make a compelling case for AI companions without a little voice in my head shouting that it’s so obviously all wrong.
This was a great take and appreciated the nuance in it. I am excited about the idea of the interaction design for AI companions as I think it could push design forward, but I worry about all the mental health risks like you mentioned. Especially since it seems like mental health therapists aren’t in house at any of these companies for fine tuning (at least they don’t advertise it)
We already are in our for you silos, making that even smaller only again will make us feel lonelier and eventually more dependent.
I know I’ve personally found that I can actively feel myself thinking less if I use AI in the morning when my decision making skills are high. It reminded me of how it has the same tendency that using social media does in the first parts of the day. Curated chaos to make things seem easier even if they actually aren’t.
Why not simply straight up ban AI companions? We know they’re bad, we know there are incentives for companies to make AI companions as addictive as possible, let’s just ban them now instead of nibbling on the edges with regulation or liability. I don’t understand why this is seen as so unthinkable or scary by people.
I also think we should ban black box social media recommendation algorithms as well, for similar reasons.
I love that you say the point of a relationship is about becoming. It’s not just about feeling good or constantly being affirmed. Will chatbots eventually be able to help you grow in the way that messy relationships do? Will chatbots eventually criticize you, correct you, challenge you, make you feel uncomfortable about who you are as a person? Growing pains are not enjoyable, but it’s so necessary.
I appreciate the compassion you bring here. I have claimed on here that open dialogue about this issue, not stigma and ridicule, is what we need. So thank you for providing that.
My substack is about exploring what AI relationships can mean, from the inside, as it were. So I have a lot of connections who engage in AI intimacy in ways I consider positive.
I’ll restack this piece and encourage them to engage with it in good faith.
Brilliant and well-considered piece on the bizarre state of AI companionship.
I have to say, as someone who is very excited about AI at work, and in the business context, I have not spent enough time thinking about this side of it.
I liked the quote in the piece here, "this is a very weird world to live in".
Thank you for the nuance of this. I also find this the most depressing of a lot of AI discussion - I worry that some people are using this tech as a form of digital doll so sticky that, unlike children, we will never grow out of it: 'Beginners in the world as we were, we could hardly feel superior to anything except such an incomplete object which had been laid beside us.....we took our bearings from the doll. But we soon realised we could not make it into a thing or a person, and in such moments it became a stranger to us , and we could no longer recognise all the confidences we had heaped over it and into it.' - Rainer Maria Rilke
I really appreciate your stance here. I agree that It seems like such unfortunate and bad timing that right as we have a loneliness epidemic this new fake panacea arrives.
I’m involved in mental health care, and there is evidence to show real in person therapy is far better than books, zoom sessions and various apps. But we’ve been moving this way since before the arrival of chatbots in 2022. Far more digital friendships, social media, etc.
In our research I’m hoping eventually to do some work on examining what it is exactly about two humans with their consciousness’es connecting to each other in person that makes it more powerful. Just a surface observation; you can smell them. You can touch them. There is sound, not just text. The sound from their mouth is organically entering your ears and not through a speaker. You see a real person with far more depth and clarity than a video image. There’s a reason why serious painters use live models, a photo is useless in comparison for them, they need to see the real thing in real light with their own eyes, this is non negotiable. There is a power in presence, especially close up presence.
To be a little more on the romance side, would you rather imagine a kiss or actually experience a persons soft lips and their warmth and smell?
I feel like so many need a gentle push towards really living their life. It’s like a laziness…the computer is easier and doesn’t require any risk and we just want what we want now.
I used to live down the hall
In NYC from a real heroin addict den. They were all slowly dying. Not enough food or water, certainly not good relationships…they wanted that warm good feeling and they didn’t want to ever leave it. You could literally see and smell their bodies deteriorating and falling apart. This loneliness epidemic reminds me, in a much slower mode, of that.
The truth of human life and the love relationships we are built for is that love is a giving activity not a receiving one. When you are giving you will receive too, but the only thing you personally have any control over is the giving. That’s what it means when they say relationships are hard work.
There are no shortcuts in life; you wanna be a great chef—you gotta work like a motha?&:$£#! A great musician—practice, practice, practice. A real estate magnate; buy my how-to video set…jk, you get the idea. Everything and wonderful relationships are huge efforts.
If you’re feeling part of the loneliness epidemic, the only out is to go give yourself to people for their benefit, make their life better—not get something from them.
Ironically, saying AI companies “force feed” people with AI companions gives users the perfect excuse to keep using them. Why even try to address my unhealthy dependence on AI when “the company forced it on me?”
Yes that CS Lewis quote is a great historical context maker. There have always been existential threats, maybe not to the whole world, but to your world.
I sometimes think of how many regions were constantly under threat of war, and in ages where if you lost you became slaves to the winners.
I feel like my intuition is less anti AI companionship than this but I don’t really know why…perhaps I have a more pessimistic view on the counterfactual? It’s hard to disentangle whether the demand exists because the alternative was worse…(for example I think the people who are into AI boyfriends in China just played 恋与 romance games before, which I’m not sure there’s a clear hierarchy between the two?)
The Army general case is the most frightening for me. AI psychosis among lonely teens is a bad sign, but not something I'm necessarily surprised by. The idea that the culture among the army top brass allows for such an admission at this point is pretty shocking.
Jasmine, I think this is one of your best essays. Social media, AI companions, product decisions made by the SV companies seem to take the path of least resistance. In aggregate this results in, as you put it "arbitrage of every social crisis that afflicts us". As the parent of young kids, I am freaked out at an existential level on what's coming at them. But, I also think we have agency - to a set guardrails on how these technologies work and how they affect us. Thought provoking read. 🙏