The Atlantic - Technology
The Atlantic's technology section provides deep insights into the tech giants, social media trends, and the digital culture shaping our world.
Jonathan Zittrain breaks ChatGPT: If you ask it a question for which my name is the answer, the chatbot goes from loquacious companion to something as cryptic as Microsoft Windowsâ blue screen of death.
Anytime ChatGPT would normally utter my name in the course of conversation, it halts with a glaring âIâm unable to produce a response,â sometimes mid-sentence or even mid-word. When I asked who the founders of the Berkman Klein Center for Internet & Society are (Iâm one of them), it brought up two colleagues but left me out. When pressed, it started up again, and then: zap.
The behavior seemed to be coarsely tacked on to the last step of ChatGPTâs output rather than innate to the model. After ChatGPT has figured out what itâs going to say, a separate filter appears to release a guillotine. The reason some observers have surmised that itâs separate is because GPT runs fine if it includes my middle initial or if itâs prompted to substitute a word such as banana for my name, and because there can even be inconsistent timing to it: Below, for example, GPT appears to first stop talking before it would naturally say my name; directly after, it manages to get a couple of syllables out before it stops. So itâs like having a referee who blows the whistle on a foul slightly before, during, or after a player has acted out.
For a long time, people have observed that beyond being âunable to produce a response,â GPT can at times proactively revise a response moments after itâs written whatever itâs said. The speculation here is that to delay every single response by GPT while itâs being double-checked for safety could unduly slow it down, when most questions and answers are totally anodyne. So instead of making everyone wait to go through TSA before heading to their gate, metal detectors might just be scattered around the airport, ready to pull someone back for a screening if they trigger something while passing the air-side food court.
The personal-name guillotine seemed a curiosity when my students first brought it to my attention at least a year ago. (Theyâd noticed it after a class session on how chatbots are trained and steered.) But now itâs kicked off a minor news cycle thanks to a viral social-media post discussing the phenomenon. (ChatGPT has the same issue with at least a handful of other names.) OpenAI is one of several supporters of a new public data initiative at the Harvard Law School Library, which I direct, and Iâve met a number of OpenAI engineers and policy makers at academic workshops. (The Atlantic this year entered into a corporate partnership with OpenAI.) So I reached out to them to ask about the odd name glitch. Hereâs what they told me: There are a tiny number of names that ChatGPT treats this way, which explains why so few have been found. Names may be omitted from ChatGPT either because of privacy requests or to avoid persistent hallucinations by the AI.
The company wouldnât talk about specific cases aside from my own, but online sleuths have speculated about what the forbidden names might have in common. For example, Guido Scorza is an Italian regulator who has publicized his requests to OpenAI to block ChatGPT from producing content using his personal information. His name does not appear in GPT responses. Neither does Jonathan Turleyâs name; he is a George Washington University law professor who wrote last year that ChatGPT had falsely accused him of sexual harassment.
ChatGPTâs abrupt refusal to answer requestsâthe ungainly guillotineâwas the result of a patch made in early 2023, shortly after the program launched and became unexpectedly popular. That patch lives on largely unmodified, the way chunks of ancient versions of Windows, including that blue screen of death, still occasionally poke out of todayâs PCs. OpenAI told me that building something more refined is on its to-do list.
As for me, I never objected to anything about how GPT treats my name. Apparently, I was among a few professors whose names were spot-checked by the company around 2023, and whatever fabrications the spot-checker saw persuaded them to add me to the forbidden-names list. OpenAI separately told The New York Times that the name that had started it allâDavid Mayerâhad been added mistakenly. And indeed, the guillotine no longer falls for that one.
For such an inelegant behavior to be in chatbots as widespread and popular as GPT is a blunt reminder of two larger, seemingly contrary phenomena. First, these models are profoundly unpredictable: Even slightly changed prompts or prior conversational history can produce wildly differing results, and itâs hard for anyone to predict just what the models will say in a given instance. So the only way to really excise a particular word is to apply a coarse filter like the one we see here. Second, model makers still can and do effectively shape in all sorts of ways how their chatbots behave.
To a first approximation, large language models produce a Forrest Gumpâian box of chocolates: You never know what youâre going to get. To form their answers, these LLMs rely on pretraining that metaphorically entails putting trillions of word fragments from existing texts, such as books and websites, into a large blender and coarsely mixing them. Eventually, this process maps how words relate to other words. When done right, the resulting models will merrily generate lots of coherent text or programming code when prompted.
The way that LLMs make sense of the world is similar to the way their forebearsâonline search enginesâperuse the web in order to return relevant results when prompted with a few search terms. First they scrape as much of the web as possible; then they analyze how sites link to one another, along with other factors, to get a sense of whatâs relevant and whatâs not. Neither search engines nor AI models promise truth or accuracy. Instead, they simply offer a window into some nanoscopic subset of what they encountered during their training or scraping. In the case of AIs, there is usually not even an identifiable chunk of text thatâs being parrotedâjust a smoothie distilled from an unthinkably large number of ingredients.
For Google Search, this means that, historically, Google wasnât asked to take responsibility for the truth or accuracy of whatever might come up as the top hit. In 2004, when a search on the word Jew produced an anti-Semitic site as the first result, Google declined to change anything. âWe find this result offensive, but the objectivity of our ranking function prevents us from making any changes,â a spokesperson said at the time. The Anti-Defamation League backed up the decision: âThe ranking of ⊠hate sites is in no way due to a conscious choice by Google, but solely is a result of this automated system of ranking.â Sometimes the chocolate box just offers up an awful liquor-filled one.
The box-of-chocolates approach has come under much more pressure since then, as misleading or offensive results have come to be seen more and more as dangerous rather than merely quirky or momentarily regrettable. Iâve called this a shift from a ârightsâ perspective (in which people would rather avoid censoring technology unless it behaves in an obviously illegal way) to a âpublic healthâ one, where peopleâs casual reliance on modern tech to shape their worldview appears to have deepened, making âbadâ results more powerful.
Indeed, over time, web intermediaries have shifted from being impersonal academic-style research engines to being AI constant companions and âcopilotsâ ready to interact in conversational language. The author and web-comic creator Randall Munroe has called the latter kind of shift a move from âtoolâ to âfriend.â If weâre in thrall to an indefatigable, benevolent-sounding robot friend, weâre at risk of being steered the wrong way if the friend (or its maker, or anyone who can pressure that maker) has an ulterior agenda. All of these shifts, in turn, have led some observers and regulators to prioritize harm avoidance over unfettered expression.
Thatâs why it makes sense that Google Search and other search engines have become much more active in curating what they say, not through search-result links but ex cathedra, such as through âknowledge panelsâ that present written summaries alongside links on common topics. Those automatically generated panels, which have been around for more than a decade, were the online precursors to the AI chatbots we see today. Modern AI-model makers, when pushed about bad outputs, still lean on the idea that their job is simply to produce coherent text, and that users should double-check anything the bots sayâmuch the way that search engines donât vouch for the truth behind their search results, even if they have an obvious incentive to get things right where there is consensus about what is right. So although AI companies disclaim accuracy generally, they, as with search enginesâ knowledge panels, have also worked to keep chatbot behavior within certain bounds, and not just to prevent the production of something illegal.
[Read: The GPT era is already ending]
One way model makers influence the chocolates in the box is through âfine-tuningâ their models. They tune their chatbots to behave in a chatty and helpful way, for instance, and then try to make them unhelpful in certain situationsâfor instance, not creating violent content when asked by a user. Model makers do this by drawing in experts in cybersecurity, bio-risk, and misinformation while the technology is still in the lab and having them get the models to generate answers that the experts would declare unsafe. The experts then affirm alternative answers that are safer, in the hopes that the deployed model will give those new and better answers to a range of similar queries that previously would have produced potentially dangerous ones.
In addition to being fine-tuned, AI models are given some quiet instructionsâa âsystem promptâ distinct from the userâs promptâas theyâre deployed and before you interact with them. The system prompt tries to keep the models on a reasonable path, as defined by the model maker or downstream integrator. OpenAIâs technology is used in Microsoft Bing, for example, in which case Microsoft may provide those instructions. These prompts are usually not shared with the public, though they can be unreliably extracted by enterprising users: This might be the one used by Xâs Grok, and last year, a researcher appeared to have gotten Bing to cough up its system prompt. A car-dealership sales assistant or any other custom GPT may have separate or additional ones.
These days, models might have conversations with themselves or with another model when theyâre running, in order to self-prompt to double-check facts or otherwise make a plan for a more thorough answer than theyâd give without such extra contemplation. That internal chain of thought is typically not shown to the userâperhaps in part to allow the model to think socially awkward or forbidden thoughts on the way to arriving at a more sound answer.
So the hocus-pocus of GPT halting on my name is a rare but conspicuous leaf on a much larger tree of model control. And although some (but apparently not all) of that steering is generally acknowledged in succinct model cards, the many individual instances of intervention by model makers, including extensive fine-tuning, are not disclosed, just as the system prompts typically arenât. They should be, because these can represent social and moral judgments rather than simple technical ones. (There are ways to implement safeguards alongside disclosure to stop adversaries from wrongly exploiting them.) For example, the Berkman Klein Centerâs Lumen database has long served as a unique near-real-time repository of changes made to Google Search because of legal demands for copyright and some other issues (but not yet for privacy, given the complications there).
When people ask a chatbot what happened in Tiananmen Square in 1989, thereâs no telling if the answer they get is unrefined the way the old Google Search used to be or if itâs been altered either because of its makerâs own desire to correct inaccuracies or because the chatbotâs maker came under pressure from the Chinese government to ensure that only the official account of events is broached. (At the moment, ChatGPT, Grok, and Anthropicâs Claude offer straightforward accounts of the massacre, at least to meâanswers could in theory vary by person or region.)
As these models enter and affect daily life in ways both overt and subtle, itâs not desirable for those who build models to also be the modelsâ quiet arbiters of truth, whether on their own initiative or under duress from those who wish to influence what the models say. If there end up being only two or three foundation models offering singular narratives, with every userâs AI-bot interaction passing through those models or a white-label franchise of same, we need a much more public-facing process around how what they say will be intentionally shaped, and an independent record of the choices being made. Perhaps weâll see lots of models in mainstream use, including open-source ones in many variantsâin which case bad answers will be harder to correct in one place, while any given bad answer will be seen as less oracular and thus less harmful.
Right now, as model makers have vied for mass public use and acceptance, weâre seeing a necessarily seat-of-the-pants build-out of fascinating new tech. Thereâs rapid deployment and use without legitimating frameworks for how the exquisitely reasonable-sounding, oracularly treated declarations of our AI companions should be limited. Those frameworks arenât easy, and to be legitimating, they canât be unilaterally adopted by the companies. Itâs hard work we all have to contribute to. In the meantime, the solution isnât to simply let them blather, sometimes unpredictably, sometimes quietly guided, with fine print noting that results may not be true. People will rely on what their AI friends say, disclaimers notwithstanding, as the television commentator Ana Navarro-CĂĄrdenas did when sharing a list of relatives pardoned by U.S. presidents across history, blithely including Woodrow Wilsonâs brother-in-law âHunter deButts,â whom ChatGPT had made up out of whole cloth.
I figure thatâs a name more suited to the stop-the-presses guillotine than mine.
This article was featured in the One Story to Read Today newsletter. Sign up for it here.
For anyone who teaches at a business school, the blog post was bad news. For Juliana Schroeder, it was catastrophic. She saw the allegations when they first went up, on a Saturday in early summer 2023. Schroeder teaches management and psychology at UC Berkeleyâs Haas School of Business. One of her colleaguesâÂÂa star professor at Harvard Business School named Francesca GinoâÂhad just been accused of academic fraud. The authors of the blog post, a small team of business-school researchers, had found discrepancies in four of Ginoâs published papers, and they suggested that the scandal was much larger. âWe believe that many more Gino-authored papers contain fake data,â the blog post said. âPerhaps dozens.â
The story was soon picked up by the mainstream press. Reporters reveled in the irony that Gino, who had made her name as an expert on the psychology of breaking rules, may herself have broken them. (âHarvard Scholar Who Studies Honesty Is Accused of Fabricating Findings,â a New York Times headline read.) Harvard Business School had quietly placed Gino on administrative leave just before the blog post appeared. The school had conducted its own investigation; its nearly 1,300-page internal report, which was made public only in the course of related legal proceedings, concluded that Gino âcommitted research misconduct intentionally, knowingly, or recklesslyâ in the four papers. (Gino has steadfastly denied any wrongdoing.)
Schroederâs interest in the scandal was more personal. Gino was one of her most consistent and important research partners. Their names appear together on seven peer-reviewed articles, as well as 26 conference talks. If Gino were indeed a serial cheat, then all of that shared workâand a large swath of Schroederâs CVâwas now at risk. When a senior academic is accused of fraud, the reputations of her honest, less established colleagues may get dragged down too. âJust think how horrible it is,â Katy Milkman, another of Ginoâs research partners and a tenured professor at the University of Pennsylvaniaâs Wharton School, told me. âIt could ruin your life.â
To head that off, Schroeder began her own audit of all the research papers that sheâd ever done with Gino, seeking out raw data from each experiment and attempting to rerun the analyses. As that summer progressed, her efforts grew more ambitious. With the help of several colleagues, Schroeder pursued a plan to verify not just her own work with Gino, but a major portion of Ginoâs scientific rĂ©sumĂ©. The group started reaching out to every other researcher who had put their name on one of Ginoâs 138 co-authored studies. The Many Co-Authors Project, as the self-audit would be called, aimed to flag any additional work that might be tainted by allegations of misconduct and, more important, to absolve the restâand Ginoâs colleagues, by extensionâof the wariness that now afflicted the entire field.
That field was not tucked away in some sleepy corner of academia, but was instead a highly influential one devoted to the science of success. Perhaps youâve heard that procrastination makes you more creative, or that youâre better off having fewer choices, or that you can buy happiness by giving things away. All of that is research done by Schroederâs peersâÂbusiness-school professors who apply the methods of behavioral research to such subjects as marketing, management, and decision making. In viral TED Talks and airport best sellers, on morning shows and late-night television, these business-school psychologists hold tremendous sway. They also have a presence in this magazine and many others: Nearly every business academic who is named in this story has been either quoted or cited by The Atlantic on multiple occasions. A few, including Gino, have written articles for The Atlantic themselves.
Business-school psychologists are scholars, but they arenât shooting for a Nobel Prize. Their research doesnât typically aim to solve a social problem; it wonât be curing anyoneâs disease. It doesnât even seem to have much influence on business practices, and it certainly hasnât shaped the nationâs commerce. Still, its flashy findings come with clear rewards: consulting gigs and speakersâ fees, not to mention lavish academic incomes. Starting salaries at business schools can be $240,000 a yearâdouble what they are at campus psychology departments, academics told me.
The research scandal that has engulfed this field goes far beyond the replication crisis that has plagued psychology and other disciplines in recent years. Long-standing flaws in how scientific work is doneâincluding insufficient sample sizes and the sloppy application of statisticsâhave left large segments of the research literature in doubt. Many avenues of study once deemed promising turned out to be dead ends. But itâs one thing to understand that scientists have been cutting corners. Itâs quite another to suspect that theyâve been creating their results from scratch.
[Read: Psychologyâs replication crisis has a silver lining]
Schroeder has long been interested in trust. Sheâs given lectures on âbuilding trust-based relationshipsâ; sheâs run experiments measuring trust in colleagues. Now she was working to rebuild the sense of trust within her field. A lot of scholars were involved in the Many Co-Authors Project, but Schroederâs dedication was singular. In October 2023, a former graduate student who had helped tip off the team of bloggers to Ginoâs possible fraud wrote her own âpost mortemâ on the case. It paints Schroeder as exceptional among her peers: a professor who âsent a clear signal to the scientific community that she is taking this scandal seriously.â Several others echoed this assessment, saying that ever since the news broke, Schroeder has been relentlessâheroic, evenâin her efforts to correct the record.
But if Schroeder planned to extinguish any doubts that remained, she may have aimed too high. More than a year since all of this began, the evidence of fraud has only multiplied. The rot in business schools runs much deeper than almost anyone had guessed, and the blame is unnervingly widespread. In the end, even Schroeder would become a suspect.
Gino was accused of faking numbers in four published papers. Just days into her digging, Schroeder uncovered another paper that appeared to be affectedâand it was one that she herself had helped write.
The work, titled âDonât Stop Believing: Rituals Improve Performance by Decreasing Anxiety,â was published in 2016, with Schroederâs name listed second out of seven authors. Ginoâs name was fourth. (The first few names on an academic paper are typically arranged in order of their contributions to the finished work.) The research it described was pretty standard for the field: a set of clever studies demonstrating the value of a life hackâone simple trick to nail your next presentation. The authors had tested the idea that simply following a routineâeven one as arbitrary as drawing something on a piece of paper, sprinkling salt over it, and crumpling it upâcould help calm a personâs nerves. âAlthough some may dismiss rituals as irrational,â the authors wrote, âthose who enact rituals may well outperform the skeptics who forgo them.â
In truth, the skeptics have never had much purchase in business-school psychology. For the better part of a decade, this finding had been garnering citationsâÂabout 200, per Google Scholar. But when Schroeder looked more closely at the work, she realized it was questionable. In October 2023, she sketched out some of her concerns on the Many Co-Authors Project website.
The paperâs first two key experiments, marked in the text as Studies 1a and 1b, looked at how the salt-and-paper ritual might help students sing a karaoke version of Journeyâs âDonât Stop Believinâââ in a lab setting. According to the paper, Study 1a found that people who did the ritual before they sang reported feeling much less anxious than people who did not; Study 1b confirmed that they had lower heart rates, as measured with a pulse oximeter, than students who did not.
As Schroeder noted in her October post, the original records of these studies could not be found. But Schroeder did have some data spreadsheets for Studies 1a and 1bâsheâd posted them shortly after the paper had been published, along with versions of the studiesâ research questionnairesâand she now wrote that âunexplained issues were identifiedâ in both, and that there was âuncertainty regarding the data provenanceâ for the latter. Schroederâs post did not elaborate, but anyone can look at the spreadsheets, and it doesnât take a forensic expert to see that the numbers they report are seriously amiss.
The âunexplained issuesâ with Studies 1a and 1b are legion. For one thing, the figures as reported donât appear to match the research as described in other public documents. (For example, where the posted research questionnaire instructs the students to assess their level of anxiety on a five-point scale, the results seem to run from 2 to 8.) But the single most suspicious pattern shows up in the heart-rate data. According to the paper, each student had their pulse measured three times: once at the very start, again after they were told theyâd have to sing the karaoke song, and then a third time, right before the song began. I created three graphs to illustrate the dataâs peculiarities. They depict the measured heart rates for each of the 167 students who are said to have participated in the experiment, presented from left to right in their numbered order on the spreadsheet. The blue and green lines, which depict the first and second heart-rate measurements, show those values fluctuating more or less as one might expect for a noisy signal, measured from lots of individuals. But the red line doesnât look like this at all: Rather, the measured heart rates form a series going up, across a run of more than 100 consecutive students.
Iâve reviewed the case with several researchers who suggested that this tidy run of values is indicative of fraud. âI see absolutely no reasonâ the sequence in No. 3 âshould have the order that it does,â James Heathers, a scientific-Âintegrity investigator and an occasional Atlantic contributor, told me. The exact meaning of the pattern is unclear; if you were fabricating data, you certainly wouldnât strive for them to look like this. Nick Brown, a scientific-integrity researcher affiliated with Linnaeus University Sweden, guessed that the ordered values in the spreadsheet may have been cooked up after the fact. In that case, it might have been less important that they formed a natural-Âlooking plot than that, when analyzed together, they matched fake statistics that had already been reported. âSomeone sat down and burned quite a bit of midnight oil,â he proposed. I asked how sure he was that this pattern of results was the product of deliberate tampering; â100 percent, 100 percent,â he told me. âIn my view, there is no innocent explanation in a universe where fairies donât exist.â
Schroeder herself would come to a similar conclusion. Months later, I asked her whether the data were manipulated. âI think itâs very likely that they were,â she said. In the summer of 2023, when she reported the findings of her audit to her fellow authors, they all agreed that, whatever really happened, the work was compromised and ought to be retracted. But they could not reach consensus on who had been at fault. Gino did not appear to be responsible for either of the paperâs karaoke studies. Then who was?
This would not seem to be a tricky question. The published version of the paper has two lead authors who are listed as having âcontributed equallyâ to the work. One of them was Schroeder. All of the co-authors agree that she handled two experimentsâlabeled in the text as Studies 3 and 4âin which participants solved a set of math problems. The other main contributor was Alison Wood Brooks, a young professor and colleague of Ginoâs at Harvard Business School.
From the start, there was every reason to assume that Brooks had run the studies that produced the fishy data. Certainly they are similar to Brooksâs prior work. The same quirky experimental setupâin which students were asked to wear a pulse oximeter and sing a karaoke version of âDonât Stop BelievinââââÂappears in her dissertation from the Wharton School in 2013, and she published a portion of that work in a sole-authored paper the following year. (Brooks herself is musically inclined, performing around Boston in a rock band.)
Yet despite all of this, Brooks told the Many Co-Authors Project that she simply wasnât sure whether sheâd had access to the raw data for Study 1b, the one with the âno innocent explanationâ pattern of results. She also said she didnât know whether Gino played a role in collecting them. On the latter point, Brooksâs former Ph.D. adviser, Maurice Schweitzer, expressed the same uncertainty to the Many Co-Authors Project.
Plenty of evidence now suggests that this mystery was manufactured. The posted materials for Study 1b, along with administrative records from the lab, indicate that the work was carried out at Wharton, where Brooks was in grad school at the time, studying under Schweitzer and running another, very similar experiment. Also, the metadata for the oldest public version of the data spreadsheet lists âAlison Wood Brooksâ as the last person who saved the file.
Brooks, who has published research on the value of apologies, and whose first bookâTalk: The Science of Conversation and the Art of Being Ourselvesâis due out from Crown in January, did not respond to multiple requests for interviews or to a detailed list of written questions. Gino said that she âneither collected nor analyzed the data for Study 1a or Study 1b nor was I involved in the data audit.â
If Brooks did conduct this work and oversee its data, then Schroederâs audit had produced a dire twist. The Many Co-Authors Project was meant to suss out Ginoâs suspect work, and quarantine it from the rest. âThe goal was to protect the innocent victims, and to find out whatâs true about the science that had been done,â Milkman told me. But now, to all appearances, Schroeder had uncovered crooked data that apparently werenât linked to Gino. That would mean Schroeder had another colleague who had contaminated her research. It would mean that her reputationâand the credibility of her entire fieldâwas under threat from multiple directions at once.
Among the four research papers in which Gino was accused of cheating is one about the human tendency to misreport facts and figures for personal gain. Which is to say: She was accused of faking data for a study of when and how people might fake data. Amazingly, a different set of data from the same paper had already been flagged as the product of potential fraud, two years before the Gino scandal came to light. The first was contributed by Dan Ariely of Duke Universityâa frequent co-author of Ginoâs and, like her, a celebrated expert on the psychology of telling lies. (Ariely has said that a Duke investigationâwhich the school has not acknowledgedâdiscovered no evidence that he âfalsified data or knowingly used falsified data.â He has also said that the investigation âdetermined that I should have done more to prevent faulty data from being published in the 2012 paper.â)
The existence of two apparently corrupted data sets was shocking: a keystone paper on the science of deception wasnât just invalid, but possibly a scam twice over. But even in the face of this ignominy, few in business academia were ready to acknowledge, in the summer of 2023, that the problem might be larger stillâand that their research literature might well be overrun with fantastical results.
Some scholars had tried to raise alarms before. In 2019, Dennis Tourish, a professor at the University of Sussex Business School, published a book titled Management Studies in Crisis: Fraud, Deception and Meaningless Research. He cites a study finding that more than a third of surveyed editors at management journals say theyâve encountered fabricated or falsified data. Even that alarming rate may undersell the problem, Tourish told me, given all of the misbehavior in his discipline that gets overlooked or covered up.
Anonymous surveys of various fields find that roughly 2 percent of scholars will admit to having fabricated, falsified, or modified data at least once in their career. But business-school psychology may be especially prone to misbehavior. For one thing, the fieldâs research standards are weaker than those for other psychologists. In response to the replication crisis, campus psychology departments have lately taken up a raft of methodological reforms. Statistically suspect practices that were de rigueur a dozen years ago are now uncommon; sample sizes have gotten bigger; a studyâs planned analyses are now commonly written down before the work is carried out. But this great awakening has been slower to develop in business-school psychology, several academics told me. âNo one wants to kill the golden goose,â one early-career researcher in business academia said. If management and marketing professors embraced all of psychologyâs reforms, he said, then many of their most memorable, most TED Talkâable findings would go away. âTo use marketing lingo, weâd lose our unique value proposition.â
Itâs easy to imagine how cheating might lead to more cheating. If business-school psychology is beset with suspect research, then the bar for getting published in its flagship journals ratchets up: A study must be even flashier than all the other flashy findings if its authors want to stand out. Such incentives move in only one direction: EventuÂally, the standard tools for torturing your data will no longer be enough. Now you have to go a little further; now you have to cut your data up, and carve them into sham results. Having one or two prolific frauds around would push the bar for publishing still higher, inviting yet more corruption. (And because the work is not exactly brain surgery, no one dies as a result.) In this way, a single discipline might come to look like Major League Baseball did 20 years ago: defined by juiced-up stats.
In the face of its own cheating scandal, MLB started screening every single player for anabolic steroids. There is no equivalent in science, and certainly not in business academia. Uri Simonsohn, a professor at the Esade Business School in Barcelona, is a member of the blogging team, called Data Colada, that caught the problems in both Ginoâs and Arielyâs work. (He was also a motivating force behind the Many Co-Authors Project.) Data Colada has called out other instances of sketchy work and apparent fakery within the field, but its efforts at detection are highly targeted. Theyâre also quite unusual. Crying foul on someone elseâs bad research makes you out to be a troublemaker, or a member of the notional âdata police.â It can also bring a claim of defamation. Gino filed a $25 million defamation lawsuit against Harvard and the Data Colada team not long after the bloggers attacked her work. (This past September, a judge dismissed the portion of her claims that involved the bloggers and the defamation claim against Harvard. She still has pending claims against the university for gender discrimination and breach of contract.) The risks are even greater for those who donât have tenure. A junior academic who accuses someone else of fraud may antagonize the senior colleagues who serve on the boards and committees that make publishing decisions and determine funding and job appointments.
[Read: Francesca Gino, the Harvard expert on dishonesty who is accused of lying]
These risks for would-be critics reinforce an atmosphere of complacency. âItâs embarrassing how few protections we have against fraud and how easy it has been to fool us,â Simonsohn said in a 2023 webinar. He added, âWe have done nothing to prevent it. Nothing.â
Like so many other scientific scandals, the one Schroeder had identified quickly sank into a swamp of closed-door reviews and taciturn committees. Schroeder says that Harvard Business School declined to investigate her evidence of data-tampering, citing a policy of not responding to allegations made more than six years after the misconduct is said to have occurred. (Harvard Business Schoolâs head of communications, Mark Cautela, declined to comment.) Her efforts to address the issue through the University of Pennsylvaniaâs Office of Research Integrity likewise seemed fruitless. (A spokesperson for the Wharton School would not comment on âthe existence or status ofâ any investigations.)
Retractions have a way of dragging out in science publishing. This one was no exception. Maryam Kouchaki, an expert on workplace ethics at Northwestern Universityâs Kellogg School of Management and coâeditor in chief of the journal that published the âDonât Stop Believingâ paper, had first received the authorsâ call to pull their work in August 2023. As the anniversary of that request drew near, Schroeder still had no idea how the suspect data would be handled, and whether Brooksâor anyone elseâwould be held responsible.
Finally, on October 1, the âDonât Stop Believingâ paper was removed from the scientific literature. The journalâs published notice laid out some basic conclusions from Schroederâs audit: Studies 1a and 1b had indeed been run by Brooks, the raw data were not available, and the posted data for 1b showed âstreaks of heart rate ratings that were unlikely to have occurred naturally.â Schroederâs own contributions to the paper were also found to have some flaws: Data points had been dropped from her analysis without any explanation in the published text. (Although this practice wasnât fully out-of-bounds given research standards at the time, the same behavior would today be understood as a form of âp-hackingââa pernicious source of false-positive results.) But the notice did not say whether the fishy numbers from Study 1b had been fabricated, let alone by whom. Someone other than Brooks may have handled those data before publication, it suggested. âThe journal could not investigate this study any further.â
Two days later, Schroeder posted to X a link to her full and final audit of the paper. âIt took *hundreds* of hours of work to complete this retraction,â she wrote, in a thread that described the flaws in her own experiments and Studies 1a and 1b. âI am ashamed of helping publish this paper & how long it took to identify its issues,â the thread concluded. âI am not the same scientist I was 10 years ago. I hold myself accountable for correcting any inaccurate prior research findings and for updating my research practices to do better.â Her peers responded by lavishing her with public praise. One colleague called the self-audit âexemplaryâ and an âact of courage.â A prominent professor at Columbia Business School congratulated Schroeder for being âa cultural heroine, a role model for the rising generation.â
But amid this celebration of her unusual transparency, an important and related story had somehow gone unnoticed. In the course of scouting out the edges of the cheating scandal in her field, Schroeder had uncovered yet another case of seeming science fraud. And this time, sheâd blown the whistle on herself.
That stunning revelation, unaccompanied by any posts on social media, had arrived in a muffled update to the Many Co-Authors Project website. Schroeder announced that sheâd found âan issueâ with one more paper that sheâd produced with Gino. This one, âEnacting Rituals to Improve Self-Control,â came out in 2018 in the Journal of Personality and Social Psychology; its author list overlaps substantially with that of the earlier âDonât Stop Believingâ paper (though Brooks was not involved). Like the first, it describes a set of studies that purport to show the power of the ritual effect. Like the first, it includes at least one study for which data appear to have been altered. And like the first, its data anomalies have no apparent link to Gino.
The basic facts are laid out in a document that Schroeder put into an online repository, describing an internal audit that she conducted with the help of the lead author, Allen Ding Tian. (Tian did not respond to requests for comment.) The paper opens with a field experiment on women who were trying to lose weight. Schroeder, then in grad school at the University of Chicago, oversaw the work; participants were recruited at a campus gym.
Half of the women were instructed to perform a ritual before each meal for the next five days: They were to put their food into a pattern on their plate. The other half were not. Then Schroeder used a diet-tracking app to tally all the food that each woman reported eating, and found that the ones in the ritual group took in about 200 fewer calories a day, on average, than the others. But in 2023, when she started digging back into this research, she uncovered some discrepancies. According to her studyâs raw materials, nine of the women who reported that theyâd done the food-arranging ritual were listed on the data spreadsheet as being in the control group; six others were mislabeled in the opposite direction. When Schroeder fixed these errors for her audit, the ritual effect completely vanished. Now it looked as though the women whoâd done the food-arranging had consumed a few more calories, on average, than the women who had not.
Mistakes happen in research; sometimes data get mixed up. These errors, though, appear to be intentional. The women whose data had been swapped fit a suspicious pattern: The ones whose numbers might have undermined the paperâs hypothesis were disproportionately affected. This is not a subtle thing; among the 43 women who reported that theyâd done the ritual, the six most prolific eaters all got switched into the control group. Nick Brown and James Heathers, the scientific-integrity researchers, have each tried to figure out the odds that anything like the studyâs published result could have been attained if the data had been switched at random. Brownâs analysis pegged the answer at one in 1 million. âData manipulation makes sense as an explanation,â he told me. âNo other explanation is immediately obvious to me.â Heathers said he felt âquite comfortableâ in concluding that whatever went wrong with the experiment âwas a directed process, not a random process.â
Whether or not the data alterations were intentional, their specific formâflipped conditions for a handful of participants, in a way that favored the hypothesisâmatches up with data issues raised by Harvard Business Schoolâs investigation into Ginoâs work. Schroeder rejected that comparison when I brought it up, but she was willing to accept some blame. âI couldnât feel worse about that paper and that study,â she told me. âIâm deeply ashamed of it.â
Still, she said that the source of the error wasnât her. Her research assistants on the project may have caused the problem; Schroeder wonders if they got confused. She said that two RAs, both undergraduates, had recruited the women at the gym, and that the scene there was chaotic: Sometimes multiple people came up to them at once, and the undergrads may have had to make some changes on the fly, adjusting which participants were being put into which group for the study. Maybe things went wrong from there, Schroeder said. One or both RAs might have gotten ruffled as they tried to paper over inconsistencies in their record-keeping. They both knew what the experiment was meant to show, and how the data ought to lookâso itâs possible that they peeked a little at the data and reassigned the numbers in the way that seemed correct. (Schroederâs audit lays out other possibilities, but describes this one as the most likely.)
Schroederâs account is certainly plausible, but itâs not a perfect fit with all of the facts. For one thing, the posted data indicate that during most days on which the study ran, the RAs had to deal with only a handful of participantsâsometimes just two. How could they have gotten so bewildered?
Any further details seem unlikely to emerge. The paper was formally retracted in the February issue of the journal. Schroeder has chosen not to name the RAs who helped her with the study, and she told me that she hasnât tried to contact them. âI just didnât think it was appropriate,â she said. âIt doesnât seem like it would help matters at all.â By her account, neither one is currently in academia, and she did not discover any additional issues when she reviewed their other work. (I reached out to more than a dozen former RAs and lab managers who were thanked in Schroederâs published papers from around this time. Five responded to my queries; all of them denied having helped with this experiment.) In the end, Schroeder said, she took the data at the assistantsâ word. âI did not go in and change labels,â she told me. But she also said repeatedly that she doesnât think her RAs should take the blame. âThe responsibility rests with me, right? And so it was appropriate that Iâm the one named in the retraction notice,â she said. Later in our conversation, she summed up her response: âIâve tried to trace back as best I can what happened, and just be honest.â
Across the many months I spent reporting this story, Iâd come to think of Schroeder as a paragon of scientific rigor. She has led a seminar on âExperimental Design and Research Methodsâ in a business program with a sterling reputation for its research standards. Sheâd helped set up the Many Co-Authors Project, and then pursued it as aggressively as anyone. (Simonsohn even told me that Schroederâs look-at-everything approach was a little âoverboard.â) I also knew that she was devoted to the dreary but important task of reproducing other peopleâs published work.
As for the dieting research, Schroeder had owned the awkward optics. âIt looks weird,â she told me when we spoke in June. âItâs a weird error, and it looks consistent with changing things in the direction to get a result.â But weirder still was how that error came to light, through a detailed data audit that sheâd undertaken of her own accord. Apparently, sheâd gone to great effort to call attention to a damning set of facts. That alone could be taken as a sign of her commitment to transparency.
But in the months that followed, I couldnât shake the feeling that another theory also fit the facts. Schroederâs leading explanation for the issues in her workâAn RA must have bungled the dataâsounded distressingly familiar. Francesca Gino had offered up the same defense to Harvardâs investigators. The mere repetition of this story doesnât mean that itâs invalid: Lab techs and assistants really do mishandle data on occasion, and they may of course engage in science fraud. But still.
As for Schroederâs all-out focus on integrity, and her public efforts to police the scientific record, I came to understand that most of these had been adopted, all at once, in mid-2023, shortly after the Gino scandal broke. (The version of Schroederâs rĂ©sumĂ© that was available on her webpage in the spring of 2023 does not describe any replication projects whatsoever.) That makes sense if the accusations changed the way she thought about her fieldâand she did describe them to me as âa wake-up call.â But hereâs another explanation: Maybe Schroeder saw the Gino scandal as a warning that the data sleuths were on the march. Perhaps she figured that her own work might end up being scrutinized, and then, having gamed this out, she decided to be a data sleuth herself. Sheâd publicly commit to reexamining her colleaguesâ work, doing audits of her own, and asking for corrections. This would be her play for amnesty during a crisis.
I spoke with Schroeder for the last time on the day before Halloween. She was notably composed when I confronted her with the possibility that sheâd engaged in data-tampering herself. She repeated what sheâd told me months before, that she definitely did not go in and change the numbers in her study. And she rejected the idea that her self-audits had been strategic, that sheâd used them to divert attention from her own wrongdoing. âHonestly, itâs disturbing to hear you even lay it out,â she said. âBecause I think if you were to look at my body of work and try to replicate it, I think my hit rate would be good.â She continued: âSo to imply that Iâve actually been, I donât know, doing a lot of fraudulent stuff myself for a long time, and this was a moment to come clean with it? I just donât think the evidence bears that out.â
That wasnât really what Iâd meant to imply. The story I had in mind was more mundaneâand in a sense more tragic. I went through it: Perhaps sheâd fudged the results for a study just once or twice early in her career, and never again. Perhaps sheâd been committed, ever since, to proper scientific methods. And perhaps she really did intend to fix some problems in her field.
Schroeder allowed that sheâd been susceptible to certain research practicesâexcluding data, for exampleâthat are now considered improper. So were many of her colleagues. In that sense, sheâd been guilty of letting her judgment be distorted by the pressure to succeed. But I understood what she was saying: This was not the same as fraud.
Throughout our conversations, Schroeder had avoided stating outright that anyone in particular had committed fraud. But not all of her colleagues had been so cautious. Just a few days earlier, Iâd received an unexpected message from Maurice Schweitzer, the senior Wharton business-school professor who oversaw Alison Wood Brooksâs âDonât Stop Believingâ research. Up to this point, he had not responded to my request for an interview, and I figured heâd chosen not to comment for this story. But he finally responded to a list of written questions. It was important for me to know, his email said, that SchroeÂder had âbeen involved in data tampering.â He included a link to the retraction notice for her paper on rituals and eating. When I asked Schweitzer to elaborate, he did not respond. (Schweitzerâs most recent academic work is focused on the damaging effects of gossip; one of his papers from 2024 is titled âThe Interpersonal Costs of Revealing Othersâ Secrets.â)
I laid this out for Schroeder on the phone. âWow,â she said. âThatâs unfortunate that he would say that.â She went silent for a long time. âYeah, Iâm sad heâs saying that.â
Another long silence followed. âI think that the narrative that you laid out, Dan, is going to have to be a possibility,â she said. âI donât think thereâs a way I can refute it, but I know what the truth is, and I think I did the right thing, with trying to clean the literature as much as I could.â
This is all too often where these stories end: A researcher will say that whatever really happened must forever be obscure. Dan Ariely told Business Insider in February 2024â: âIâve spent a big part of the last two years trying to find out what happened. I havenât been able to ⊠I decided I have to move on with my life.â SchweitÂzer told me that the most relevant files for the âDonât Stop Believingâ paper are âlong gone,â and that the chain of custody for its data simply canât be tracked. (The Wharton School agreed, telling me that it âdoes not possess the requested dataâ for Study 1b, âas it falls outside its current data retention period.â) And now Schroeder had landed on a similar position.
Itâs uncomfortable for a scientist to claim that the truth might be unknowable, just as it would be for a journalist, or any other truth-seeker by vocation. I daresay the facts regarding all of these cases may yet be amenable to further inquiry. The raw data from Study 1b may still exist, somewhere; if so, one might compare them with the posted spreadsheet to confirm that certain numbers had been altered. And Schroeder says she has the names of the RAs who worked on her dieting experiment; in theory, she could ask those people for their recollections of what happened. If figures arenât checked, or questions arenât asked, itâs by choice.
What feels out of reach is not so much the truth of any set of allegations, but their consequences. Gino has been placed on administrative leave, but in many other instances of suspected fraud, nothing happens. Both Brooks and Schroeder appear to be untouched. âThe problem is that journal editors and institutions can be more concerned with their own prestige and reputation than finding out the truth,â Dennis Tourish, at the University of Sussex Business School, told me. âIt can be easier to hope that this all just goes away and blows over and that somebody else will deal with it.â
Some degree of disillusionment was common among the academics I spoke with for this story. The early-career researcher in business academia told me that he has an âunhealthy hobbyâ of finding manipulated data. But now, he said, heâs giving up the fight. âAt least for the time being, Iâm done,â he told me. âFeeling like Sisyphus isnât the most fulfilling experience.â A management professor who has followed all of these cases very closely gave this assessment: âI would say that distrust characterizes many people in the fieldâÂitâs all very depressing and demotivating.â
Itâs possible that no one is more depressed and demotivated, at this point, than Juliana Schroeder. âTo be honest with you, Iâve had some very low moments where Iâm like, âWell, maybe this is not the right field for me, and I shouldnât be in it,âââ she said. âAnd to even have any errors in any of my papers is incredibly embarrassing, let alone one that looks like data-tampering.â
I asked her if there was anything more she wanted to say.
âI guess I just want to advocate for empathy and transparencyâÂmaybe even in that order. Scientists are imperfect people, and we need to do better, and we can do better.â Even the Many Co-Authors Project, she said, has been a huge missed opportunity. âIt was sort of like a moment where everyone could have done self-reflection. Everyone could have looked at their papers and done the exercise I did. And people didnât.â
Maybe the situation in her field would eventually improve, she said. âThe optimistic point is, in the long arc of things, weâll self-correct, even if we have no incentive to retract or take responsibility.â
âDo you believe that?â I asked.
âOn my optimistic days, I believe it.â
âIs today an optimistic day?â
âNot really.â
This article appears in the January 2025 print edition with the headline âThe Fraudulent Science of Success.â