AI Cannot Write (Part 1)
Revisiting an idea about generative AI and human writing from a year ago.
About a year ago, I joined many academics in pondering what generative AI meant for teaching and scholarship. I also started thinking about what my field—rhetoric, composition, and writing studies (RCWS)—might offer to help the world make sense of this technology. I wrote some material (first a Facebook post, then an essay I never published) that made the following claim: AI cannot write. What follows are some of my original thoughts from early 2023, including excerpts from a discussion with ChatGPT (3.5). In part 2, I will update my conversation with ChatGPT (4.0) to consider how it may have changed, and what that might mean for how we think about writing.
The discipline of RCWS has spent the better part of seventy years inquiring into the nature of writing and how to teach it. With nearly every concept, AI fails the test.
I will only represent AI as cake from now on. Image produced in Adobe Express.
We know that writing emerges across disciplinary, professional, and public domains because of some social purpose, often a pressing one. For example, commentary on AI itself exists because we humans have a need to understand how our lives and our careers may be affected by a technology that resembles (some of) us (some of the time).
However, ChatGPT and other large language models (LMMs) do not have such purposes. When I asked ChatGPT how it understands the purpose of a given prompt, it responded:
ChatGPT understands the purpose of a prompt through pattern recognition and its training on a diverse range of text examples. During its training, it was exposed to a large corpus of text data which included various writing styles and purposes. As a result, it can identify common patterns and understand the purpose of a prompt based on the context and language used. However, it's important to note that the accuracy of its understanding can depend on the quality and clarity of the prompt, and it may not always be able to accurately determine the purpose.
In other words, ChatGPT’s output is built on probability, not purpose. Given the context of a prompt and the corpus on which it was trained, an LMM will likely produce a range of responses, but such responses can only be intelligible, accurate, and useful within the context of human-driven purpose.
RCWS has also demonstrated that there is no one definition of “good writing.” All readers’ and writers’ understanding of “good” derives in part from our participation in communities of practice such as academic disciplines. Currently, AI has no community of practice. The large corpora used to train LMMs like ChatGPT currently do not include proprietary and paywalled information that circulates in communities of practice. But more importantly, platforms like ChatGPT struggle to distinguish amongst types of writing used by communities of practice, especially kindred ones. For example, when I asked it to distinguish between writing in mathematics and physics, it responded:
Writing in mathematics tends to focus on precise and concise statements of abstract concepts and their relationships, often with a heavy use of symbols and formal definitions. On the other hand, writing in physics often involves describing experimental observations and developing models to explain those observations in terms of physical laws and concepts. While mathematics writing may also involve problem-solving and derivations, physics writing also includes discussions of the real-world implications and applications of the concepts being studied. Additionally, physics writing often involves a greater emphasis on clear communication, as the concepts and models being described may have practical implications and be used by a wider audience than mathematical writing.
No doubt, many mathematicians would bristle at the notion that their writing is any less clear than a physicist’s. Notions of clarity, after all, are a matter of readers’ interpretations. What may seem opaque or impractical to one reader may be perfectly intelligible to another.
Furthermore, a reader’s willingness to stick with a difficult piece of writing depends in part on the writer’s ethos. The more we attribute credibility to a writer, the more we will work through difficult prose or unclear ideas. Here again, AI fails. Its tendency to introduce errors has been well documented. NPR reporters found AI platforms singularly incapable of basic rocket science; a colleague of mine got Chat GPT to say that Joe Biden was the first Jewish president; and journalists are sounding the alarm on the potential for AI to add yet more dis- and misinformation into our media ecology.
But ethos hinges on more than simple factual credibility. It also derives from our sense of a writer’s experience, expertise, and intention (which again depend a lot of communities of practice). ChatGPT admitted its limitations when I asked it this question:
As an AI, I do not have personal credibility, expertise, or experience like a human does. I am a machine learning model that generates text based on patterns learned from a large dataset of text. Therefore, my "ethos" is based on the credibility, expertise, and experience of the people who wrote the text in the dataset I was trained on.
Regarding intention, I am programmed to respond to a prompt and answer questions, but I don't have personal intentions or motivations like humans do. However, I can be programmed to write text that simulates the intention of a specific person or perspective, by being trained on a dataset of text that represents that perspective.
It is important to note that while my text generation may be influenced by the ethos of the data that I was trained on, it is not the result of personal credibility, expertise, or experience, or personal intentions.
Tellingly, ChatGPT recognizes that any ethos it has comes from its human forebears, those otherwise unacknowledged people who wrote its corpus—and, I would add, those who programmed it. Anything else, from intentions to signature styles, are mere simulations.
Finally, RCWS has long recognized writing as epistemic—that is, as an activity that not only records, but also generates, knowledge. In communities of practice like academic disciplines, novelty is a core value: we seek out instances of writing that challenge existing ideas or provide fresh insights into tricky phenomena. But built as it is exclusively on what came before, ChatGPT and other LMMs struggle to produce truly novel ideas, even if the specific words on screen have never been written that way before.
Lest all this sound like a mere “gotcha” moment, there are very real implications for both our attitudes towards AI, and how it might affect higher education. We need not simply resign ourselves to a fully automated future. We can continue to insist on, and to teach, the human elements of knowledge making, interpretation, expertise, and experience.
The more intentional we are about sharing not only what we do in our respective fields, but also why, the more our students will be able to understand the purposes that communities of practice bring to their communication. Our students can decide for themselves—or, ideally, alongside us—to take up those purposes, or to inject their own. But we must give them the opportunity. If we continue to assign writing without actually teaching it—teaching its purposes, the ways writers in various fields craft their ethos, or build knowledge through expert use of evidence, analysis, and domain knowledge—then students will focus more on producing products that look polished but actually do not accomplish any of the purposes we may wish to withhold for actual humans.
Unfortunately, the current reward systems in many institutions of higher education do not allow room for the kind of deep pedagogical and curricular re-tooling this teaching will require. If anything, the system currently rewards spending less time on active learning, meaningful and intentional assignment design, and new forms of assessment that center students’ thinking, and not just their product-making.
If higher education is going to continue to be a place that centers thinking, meaning-making, and deep learning for our students, we will need to see more rewards for scholarly teaching and the scholarship of teaching and learning, along with structural changes like smaller class sizes. Without these and other changes, we are likely heading to a future where we continue to struggle against AI, rather than work with it towards higher goals.
I still believe what I wrote then: that we need to teach writing, not just assign it, which requires teaching the human elements of knowing, doing, and being that inform our writing. And I also continue to believe that higher education will need structural changes for that to happen. To put it succinctly: faculty will need to get really good at teaching if they are to survive what may be coming.
In my next post, I will revisit the question of whether or not AI can write in an updated conversation with ChatGPT.
In the meantime, if this post got your ideas turning, consider contributing to Bad Ideas about AI: Toward Generative Practices for Teaching, Learning, and Communication, co-edited by Anna Mills, Mandy Olejnik, Miranda Rodak, Shyam Sharma, and me. Learn more at https://www.tinyurl.com/badideasbook.
Do we even need brick and mortar to correct the trajectory academia?
FYI, you wrote LMM instead of LLM a few times. Typo?