I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
At a minimum it’s probably more accurate than “AI”.
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
Meanwhile, the paper has 2 points of criticism towards AI. 1 is a bunch of carbon consumption complaints assuming NVIDIA cards with coal-fired power, while a lot of effort at contemporary Google went towards getting TPUs running on green power. I suspect this was what people wanted to object to, a lot of effort went into those green power projects and she was just denying it. The complaint seems prophetic now but it was not true about Google then.
The other criticism was about which language the LLMs use, they average the input data of normal humans instead of talking the way the paper author thinks they should talk. The phrase "women doctors" is called out as problematic. I'm less inclined to think people objected strongly to this given the zeitgeist at the time, it was probably people who worked on the green energy projects and were pissed off that their contributions were ignored, but still, nobody elected her Queen of English, she can have her opinions but she's not a victim for not having them adopted by everyone.
I think she's since since lost a lot of her allure, especially when she didn't change her mind when the facts about the AI water usage changed 1000x
I'd say what's under debate is whether uncritical LLM adoption is mainly unethical or mainly religious.
I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure.
I appeciate short letters like this that get straight to the point...
Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition.
> The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.
Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are.
[1] Coined by Marvin Minsky: https://www.thekurzweillibrary.com/consciousness-is-a-big-su...
Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding.
[1] https://chrisclay.substack.com/p/what-is-superposition-in-ne...
There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
Now if they had said, "Imagine your average American ..." (/s)
So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
They never are. Ever.
It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence.
(fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it.
So it's a great phrase because in just about every case I can ignore what someone says afterwards.
Similar to "glorified autocomplete."
Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
- Lots of Haiku around, many mistakes unless process is very clear - Some Sonnets, still do mistakes but can adapt - Some Opus, able to improvise and think outside the box.
But even the Human Opus/Mythos are hilariously wrong sometimes.
Did she really think Google cares about ethics? Such positions seem purely performative, we all know that ethics go out the window first to make room for more profits.
>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.
This is Google's side of it; I think the following is a fair piece of primary-source journalism if you want to go deeper:
https://www.platformer.news/the-withering-email-that-got-an-...
Is it possible you're making the following error described in the article?
> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?
I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
(BTW, quite bold to say input data from Reddit and 4Chan is how “normal” people speak. There is a lot of language in the training data of any model you really do not wish your application to use ever.)
She was well known to be toxic and extremely exploitive of victim privilege
To me the real question begins only once we have a clear example of a non-trivial scientific discovery that is implicit (IE, not an obvious outcome of reading the literature and talking to the experts) and experimentally verifiable. Once that happens- especially if it is a reproducible process (IE, more discoveries) and it's significant (IE, impacts human life and mind in some profound way)- then the onus very much lies on Bender and her coauthors to explain whether we need more than a sufficiently advanced stochastic parrot.
This question depends on how you define research productivity. There is close to two hundred AI papers published every weekday. Most of them are about GenAI. Most don't seem to be all thay good. The progress in actual model improvement had mostly stalled. If you interact with the latest "raw" models they display all of the issues we've seen in GPT-3.5, just at a smaller rate. The "amazing gamechanger breakthroughs" I read about on social media every week do not seem to lead anywhere. It's all kind of boring, really.
The new "hotness" in AI is clearly building more and more elaborate harnesses. This is not at all the direction AI boosters have predicted couple years ago.
Personally, I think the "stochastic parrot" mental model is far more useful for science, because it primes people for proper testing, skepticism and researching alternatives. If you want useful AI, you want people working on it being skeptical, not credulous.
By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god.
People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written.
After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features.
There has been a lot of subsequent research that really calls many of the statements in this article into question.
At this point, I think the authors are really counting on people not to read their paper.
2. LLMs are detemrinistic. They have a parameter to tune how stochastic they are.
You can't. That's just the way the news goes.
Essentially, what you are saying is that because some people somewhere frame a statement as pejorative, the statement itself is inherently pejorative. By that logic, every criticism ever articulated is inherently an insult.
From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective.
I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
Where can I access such a Lisp expert system?
If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
There's a lot more happening behind the scenes when a human repeats phrases than what's happening in an LLM.
Sociological phenomenon. The desire to be liked, successful, or popular. The feeling that those phrases brings up.
LLMs are not experiencing any of that. As far as we know, neither is a parrot.
> What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go?
And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders.
Humans learn language opportunistically. Toddlers start with a powerful "superchimpanzee" understanding of the real world, and use that to learn words in order to satisfy their needs and desires. Statistical frequency is incidental to what words a toddler learns: what matters is the real-world context. Also note how important it is that infants instinctively understand nonverbal communication.
The most depressing thing about the 2020s AI summer is watching ignorant tech workers use the success of LLMs to launder their own ignorant misanthropy. Your views are many many centuries out of date.
But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?
The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
Exhibit A.
Isn't that pattern matching essentially?
The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.
From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:
drop $thing from skyscraper --> bad behavior
drop $thing --> physics
can of $stuff --> contents in/out of can
It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?
It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.
I believe you are projecting something that is not there onto a completely mindless stochastic process.
Also to compare growing food with the totally optional, not useful in the slightest, LLMs that somehow demand local populaces bend to their will for reasons that never seem to benefit them is just bonkers level of self-blinding when it comes to populations absolutely despising big tech, big tech leadership, and big tech practices.
This mania might finally cause the software industry to become a highly regulated with licenses similar to that of other engineering disciplines due to amount of optional destruction they have decided to unleash upon on the planet in such a short time frame.
I’m absolutely positive St. Augustine wasn’t some prodigious person who was able to self-introspect his early language learning and break past infant amnesia. If we’d have any means on verifying, my bet would be on that he just made this all up, post-hoc rationalizing all of that.
He can remember some sensory and pre-language mental states (desire to achieve some other state, like change of body position, or experience a taste, or a different view), but all that “whimpering and grunts and various gestures of my limbs” rationalized as purposeful communication is pure fiction. At least to my understanding, of course.
> Also note how important it is that infants instinctively understand nonverbal communication.
What does “understand” here means, and what infants exactly?
From what we know about human development, before somewhere between 2 and 9 months of age infants are unable to purposefully communicate in principle. No recipient in mind: they don’t track gaze, they don’t point at objects, they don’t share affect. Purely reflexively they can cry when they hear others crying (just like how we yawn with others), but that’s about it - just a resonance, not communication.
Obviously, comparing two different machineries suggests us they’re different (duh!) Humans aren’t LLMs, and language acquisition happens significantly different between two. But stochastic parrot is not an animal, it’s a basic principle, and that principle may possibly hold true to all life with associative learning capabilities.
And this is a reply to her comment about water usage where it becomes clear she's not arguing in good faith. https://x.com/AndyMasley/status/1990498830131888173
See? Middle-Eastern investors are growing alfalfa in the western desert using legal allotments of water! That is so much worse than what we’re doing! Go after them!
They can both be using an egregious amount of water for silly purposes.
The other part of the water debate is also the pollution different systems create. Many data centres went in with the promise of closed-loop systems but changed half-way through construction and couldn’t be stopped.
I think it’s more complicated than, “they’re wrong, it’s just hype.”
I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
So personally I would still rather undersell it. But parrot or not, it is still a terribly useful little bird indeed.
> "Stochastic" means having a random pattern or variable that can be analyzed statistically but not predicted precisely
Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware.
For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.
Your argument makes sense if ai datacenters were using something close to like alfafa farming but the difference between them is soo massive it does not make sense.
Reducing pollution is a much better problem to fight for
So that's why "Brave minority woman unfairly fired by evil AI corporation" sells better than "Self-entitled minority woman is terrible bully to colleagues"
LLM’s are usually unexpected only when they malfunction and sprout same letter again and again etc - hardly a literary masterpiece. They make very easily recognisable patterns that we can use as helpful tools, but in the end they are devoid of any meaning apart from what we give them. Of course one could say same about art and all language, but I think there still is the fact that we apes somehow recognise each other. And besides, we do know the internal functions that drive the parroting. It is admittedly bit tricky, but in no way as magical as people purport it to be.
As far as prediction - I mean sure the cortex and LLMs do prediction, but then so can RNNs or diffusion models or any other generative model. Really any ML architecture is learning to compress its environment in pursuit of modelling. More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?
Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?
In March 2021, a group of four researchers—a collaboration of linguists and computer scientists—published their now legendary paper “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜”
The paper received significant attention at the time (in part because Google fired two of the authors, Timnit Gebru and Margaret Mitchell, shortly before its publication). It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension. And over the past five years, the analogy has spread well beyond the academic field where it originated, spawning debates and inspiring projects such as a shoulder-mounted robot named the Stochastic Parrot.
But that wider usage has also led to misconceptions about what the phrase originally meant. Lead author Emily M. Bender, a professor of computational linguistics at the University of Washington, recently wrote a blog post to debunk common misconceptions about the paper on its five-year anniversary.
Bender spoke with IEEE Spectrum about these misconceptions, the field of computational linguistics, and the current discourse around artificial intelligence.
How would you describe your work as a computational linguist?
Emily M. Bender: Linguistics, very generally, is the study of how language works and how we work with language. I contribute to that, and I also work in computational linguistics, training students who are going to go on to build language technology.
Language technology actually stands alone as valuable and interesting, independent of whether or not someone wants to use it for their project of artificial intelligence. Language technology includes things like automatic transcription, machine translation, spell check. And a lot of the work that I do personally, when I am building things, has to do with building machine-readable, but also human-readable grammars that model linguistic phenomena in different languages. That’s about using computers in the service of linguistic hypothesis testing.
You’ve argued that the term “artificial intelligence” obscures more than it clarifies. Why?
Bender: Many reasons. I think that it makes it difficult to actually have good discussions about technology and make wise decisions about it, if the way we’re talking about it doesn’t make clear what the technology is. The phrase “artificial intelligence” both groups together disparate technologies and oversells what each one of them can do. So if we are trying to decide whether or not to use something, how to regulate something, we are much better off with clearer descriptions.
In general conversation, AI has become almost synonymous with “chatbots” or “LLMs.” Is that a problem?
Bender: For many people, they’ll say, “I use it to do blah blah blah.” So what do you mean by “it”? And then they’ll say, “Oh, I mean Claude” or ChatGPT or Gemini, so they are talking about these chatbots. But then other people will say, “You can’t say AI is all bad, because what about AlphaFold?”
So, yes, for many people, they are talking about chatbots built on top of large language models, but [they’re] also not really clear that those things are separate from something like AlphaFold. And when we have news reporting that says “scientists use AI to discover a new drug,” well, what did they use? If what they’re talking about is something much more narrow, maybe it’s protein folding, maybe it’s some other kind of statistical modeling [like in] weather modeling. That’s a very different kind of technology than ChatGPT.
Do you think there’s a value to an umbrella term like “artificial intelligence”?
Bender: Well, there’s a value to people who are trying to sell this—so too the tech companies trying to raise their valuations. Also, the way research funding is set up right now, it is very hard to get funded if you don’t call what you’re doing artificial intelligence. That I think is a net negative, but for any individual trapped in that system, that can have value in the moment.
What are the most common misconceptions about the “stochastic parrots” metaphor?
Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.” That paper was written in late 2020. We were talking about large language models. I’m pretty sure the word AI comes up only once at the very end, and that’s talking about how, if you’re going to develop systems that are meant to do things like what people do, you have to be very careful that you are not creating something that can be mistaken for a person. The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
So in the paper, toward the very end, we sort of generalized to AI. But the phrase “stochastic parrots” specifically refers to large language models, and the phrase “artificial intelligence” refers to many different things. So we were never claiming that a chess engine or AlphaFold or an image labeling system or a machine translation system, any of those things that are sometimes called artificial intelligence, are stochastic parrots. We were specifically talking about using large language models to produce synthetic text.
Another one is that “stochastic parrot” got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way, but that’s not how I intended it, because it’s just a description of what these systems actually are. To see it as an insult requires either the belief that the large language model is the kind of thing that can take offense, which it isn’t, or that these large language models should be understood as steps toward this grand ideal that I don’t hold of artificial intelligence.
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
RELATED: The Great Chatbot Debate: Do They Really Understand?
For readers who don’t know, the “octopus test” comes from a 2020 paper that imagined an octopus recognizing the statistical patterns within messages passed through an undersea cable. With the octopus test and stochastic parrots, you’ve used animal metaphors a couple of times now. Is that intentional?
Bender: No, it’s not intentional. With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at. But also, octopuses are just inherently funnier.
I was looking back at that paper and was surprised that the term “stochastic parrots” actually only appears twice in the text itself. Why did you include it in your title?
Bender: Because we liked it! And a catchy title is good self-marketing of an academic paper. The reason that there’s not so much of it in the paper is that we were really looking at the full range of risks of making language models ever bigger. The phrase “large language model” also doesn’t show up in the paper, because people weren’t talking about them that way.
So the section on synthetic text, in some ways it felt like we were on thin ice, because at that point in time it was hard to imagine that anybody would want synthetic text. That part of the paper became much more relevant when OpenAI imposed ChatGPT on the world. Then that particular part of the paper comes out as important. But we also talk about environmental impact. We talk about the ways in which these systems will absorb the biases of their training data. We talk about how the training data is never collected well. There’s a lot of various points in there, and the issues about synthetic text were just one.
Researchers at MIT Media Lab created a Stochastic Parrot robot as a response to the observation that many chatbots tend to be sycophantic, or overly agreeable. Does that trend relate to the dangers you laid out in your paper?
Bender: When we wrote that paper in late 2020, at the time, people were not super excited about synthetic text, nor about chatbots. Chatbots had been around. We had Weizenbaum’s Eliza in the 1960s, and then the very annoying automatic customer service systems that have gotten much more fluent with the large language models, and no less annoying.
So, that was the state of things. OpenAI had put out GPT-2 and GPT-3 for people to play with, and you could get them to extrude synthetic text, but the chat interface hadn’t been wrapped around those yet. We also hadn’t seen the layers of additional training that lead to the behavior that’s interpreted as sycophantic. The reason that you get the chatbot saying, “Oh, that’s a good idea,” or if you say you’re wrong, it says, “Oh, I’m so sorry, you’re right,” that kind of response has to do with additional layers of training past the original pre-training.
What do you wish more people understood about language models?
Bender: The message that I always bring when I have a chance is that, when the text that comes out of one of these systems makes sense, it’s because we are making sense of it. This is also in the stochastic parrots paper. Anytime we are evaluating this kind of technology, we have to account for our ability to make sense of language and keep that in view as we are deciding what’s going on with the technology. That is frequently lost in these discussions.
If you were to redo or update the stochastic parrot paper now, is there anything that you would change about it?
Bender: There was one really big form of harm that we did not cover in the paper, and that has to do with exploitative labor practices. Under that, I include both the horrible conditions that many data workers face, and also the massive theft of people’s creative and intellectual output that underlies these systems. Those issues should have been included in the paper. It’s not that they were unknown in the world then, but they didn’t make it into what we surveyed, and should be there.
This story was updated on 1 July 2026 to clarify the research areas of the stochastic parrots paper authors.