Disclaimer
This article is based on the paper “Pretraining Data Can Be Poisoned through Computational Propaganda,” which was published on arXiv on July 16, 2026.
The paper is a preprint that has not yet completed peer review. It does not prove that any specific commercial AI model currently in operation has been poisoned through this method.
What This Article Covers
What Is Coordinated Online Information Manipulation?
The term computational propaganda, which appears in the paper’s title, refers to the organized use of bots, automation, algorithms, and repeated posting to spread specific claims or information online.
This is different from an individual posting false information once or twice.
It can involve using multiple accounts and automated tools to repeatedly expose users to the same claim or distribute similar posts across many websites at the same time.
For clarity, this article refers to such activity as coordinated online information manipulation.
Until now, this type of activity has primarily been discussed as an attempt to influence human judgment and public opinion.
However, a new risk has emerged as publicly available web content has become a source of training data for AI models.
When attackers repeatedly distribute particular claims or false information online, the content may not only be read by people. It may also be collected by web crawlers and eventually read by future AI models.
This means that the targets of coordinated online information manipulation may expand from people to AI training systems themselves.
Attackers Do Not Need to Hack an AI Company Directly
When people think about data poisoning, they often imagine an attacker directly breaking into an AI company’s servers or training dataset.
The attack examined in this study works differently.
The attacker does not need direct access to an AI company’s internal systems or training data repository.
Instead, the attacker can use web features that allow third parties to publish content, such as comment sections, discussion boards, and public forums.
The attack could unfold as follows:
- An attacker posts a specific claim or false information across multiple websites.
- A web crawler collects those pages.
- The page body and comments are extracted as text.
- The extracted documents pass through language detection and quality filters.
- Some of the manipulated content survives and enters the AI pretraining dataset.
- The trained model may later repeat the targeted claim or show a preference for a particular subject when answering similar questions.
The defining feature of this attack is that it does not directly target the completed AI dataset. Instead, it manipulates the web environment from which the dataset is built.
In other words, the target is not the finished training dataset but the information ecosystem that exists before the dataset is created.
Comments Can Also Enter AI Training Data
A web page does not contain only the article or content written by its operator.
It may also display comments, customer reviews, discussion-board replies, public profiles, and other forms of user-generated content.
People can usually distinguish the main article from the comments below it. Web crawlers and text-extraction systems, however, do not always make that distinction clearly.
When comments are included in a page’s HTML as ordinary text, a crawler may collect them along with the main content.
The researchers analyzed more than 180,000 web pages obtained through Common Crawl.
They found signs of comment platforms on approximately 3.4% of the pages.
After inserting experimental content into pages with comment sections and reproducing the text-extraction process, the researchers found that approximately 71.9% of the inserted comments remained in the extracted text.
After language detection, document-quality assessment, repetition removal, and other data-cleaning stages were applied, the estimated proportion that would ultimately enter the training dataset was approximately 0.13% of all insertion attempts.
A rate of 0.13% may appear small.
However, AI pretraining datasets can contain billions of documents and enormous volumes of text.
At that scale, even a low inclusion rate can translate into a substantial number of documents.
The important question is not whether every piece of manipulated content survives.
The key question is whether large-scale automated distribution can allow at least some manipulated content to reach the final training dataset.
What Does HALFLIFE Measure?
The researchers proposed an analytical framework called HALFLIFE to examine how manipulated content can survive until it reaches the final training dataset.
HALFLIFE divides the process into three stages and measures the likelihood of passing each one.
1. Can Content Be Inserted into the Web Page?
The first stage examines whether a web page includes a comment section or public posting feature that allows an outside user to add content.
Even without administrative access to the website, an attacker may satisfy this condition by using a publicly available comment function.
2. Does the Crawler Collect the Inserted Content?
The second stage examines whether comments displayed on the page are included in the HTML and in the resulting crawl data.
Some content may be visible to a person but loaded separately through a script, meaning that a conventional crawler may not collect it.
3. Does the Content Survive Data Cleaning?
The final stage examines whether the collected content passes through language detection, quality assessment, deduplication, and other filtering processes.
If the content fails at any one of these stages, it will not enter the final training dataset.
However, even if only a small proportion passes all three stages, large-scale automated posting may still create an opportunity to influence the resulting data.
The central idea behind HALFLIFE is that the analysis should not stop at whether manipulated content exists online. It must determine whether that content can survive the entire data-processing pipeline.
Does Poisoned Training Data Actually Affect the Model?
The presence of content in a training dataset does not necessarily mean that it will affect the model.
The researchers therefore conducted separate controlled experiments to determine whether the model’s responses would also change.
They trained models ranging from 65 million to 1.3 billion parameters while assigning 0.001%, 0.01%, or 0.1% of the total training tokens to content designed to promote a preference for a specific company or product.
The results showed that pretrained base models became more likely to favor the target selected by the attacker.
When supervised fine-tuning was applied afterward, the poisoning effect decreased, particularly in the larger models.
However, the effect did not disappear completely under every experimental condition.
The researchers also removed conspicuous conversational labels such as User and Assistant and rewrote the content as ordinary questions and answers or natural prose.
Even in those formats, the content affected the preferences expressed by the base models.
This suggests that detecting and removing only certain phrases or recognizable conversation formats may not be enough to prevent data poisoning.
When manipulated content is written naturally and resembles ordinary web content, existing quality filters may classify it as legitimate material.
Not All Web Content Provides the Same Attack Route
The fact that third-party content appears on a website does not mean that all such content can be used to poison AI training data.
The researchers examined programmatic advertising as well as comment sections.
Advertisements are also third-party content displayed on web pages. However, many ads are loaded through separate scripts after the page opens.
As a result, advertising text may not remain in the ordinary text collected through conventional DOM-based crawling and extraction.
Under the conditions of this experiment, the researchers concluded that programmatic advertising was unlikely to provide an effective route for text-based data poisoning.
What matters is not simply whether people can see content on a web page.
The entire pipeline must be examined to determine whether the content is collected by a crawler, extracted as text, and retained after data cleaning.
Comments were more likely than advertisements to remain in the training-data pipeline because they are often embedded in the page’s HTML alongside the main content.
Why Can Existing Data Filters Miss Manipulated Content?
Current training-data pipelines often assess the overall quality of a document.
They may check whether a document is too short, contains excessive symbols, repeats the same sentences, uses the intended language, or resembles natural and useful writing.
However, a single web page can contain content from several different sources.
- The main article written by the website operator
- Comments written by registered users
- Posts written by anonymous users
- Repeated sentences generated by automated bots
When the main document is high quality, the entire page may pass through a filter even if some of its comments are problematic.
Manipulated content can be even more difficult to distinguish when it is grammatically natural and appears relevant to the page’s topic.
Attackers may also express the same claim in multiple ways, making simple deduplication less effective.
Future data-cleaning systems therefore cannot rely only on deciding whether a document is good or bad.
They must also determine who created each section of the document and how that content was added to the page.
AI Security Now Extends Across the Entire Data Supply Chain
From an IT systems perspective, the central issue raised by this study is AI data supply chain security.
AI training data is not simply a collection of files downloaded from the internet.
It is the result of multiple systems and processes, including web crawling, HTML analysis, text extraction, language detection, quality assessment, deduplication, and dataset mixing.
Organizations therefore need to track which content entered or was removed at each stage.
1. Separate Different Types of Page Content
The main article, comments, reviews, discussion-board posts, and other user-generated areas should be extracted separately and evaluated under different filtering rules.
Content written by a site operator should not automatically receive the same trust level as an anonymous comment.
Comments and other user-generated content may need stricter evaluation or placement in a separate data category.
2. Manage Data Provenance
Organizations should record more than the source URL.
They should also retain the collection date, page section, apparent content author, extraction tool, and filtering results.
This is known as data provenance or data lineage.
When data lineage is available, organizations can trace which datasets and models may have been affected after a problem is discovered.
If coordinated manipulation is later identified at a particular source, data from that source can be located, removed, or reviewed again.
3. Analyze Changes Over Time
Different crawls of the same page should be compared to determine whether large numbers of comments or sentences appeared during a particular period.
When similar claims suddenly increase across multiple sites, organizations may need to investigate whether they are part of a coordinated distribution campaign.
A single document may appear natural in isolation. Abnormal activity may become visible only when temporal changes and cross-site posting patterns are analyzed together.
4. Detect Coordinated Posting Patterns
When similar sentences or claims appear across many sites at the same time, organizations should consider whether they may be connected to the same manipulation campaign.
The analysis should examine relationships and repetition patterns across documents, rather than assessing each document independently.
Account creation dates, posting intervals, sentence structures, and link-sharing patterns may also help identify abnormal behavior.
5. Strengthen Defenses at the Web Platform Level
Website operators can also strengthen anonymous-comment restrictions, posting-rate limits, identity verification, automated-post detection, and content moderation.
Preventing large volumes of manipulated content from being posted is not only a website-management issue. It may also help protect the reliability of future AI training data.
Data poisoning is therefore not a problem for AI companies alone.
It is a supply chain problem involving web platforms, content operators, crawling providers, and AI developers.
Data Provenance May Matter More Than Data Volume
AI development has often been associated with collecting more data to improve model performance.
However, increasing the amount of collected data can also increase the amount of contaminated or manipulated content.
Coordinated online information manipulation does not stop after one false claim is published.
Automated accounts and content-generation tools can repeat the same claim across multiple sites and express it in many different forms.
When the wording changes slightly, conventional deduplication systems may have difficulty identifying the repetition.
More data also does not necessarily mean more independent information.
When a single claim appears in thousands of posts, the dataset may make it look as though many independent sources support the same position.
In reality, the content may have originated from one coordinated campaign.
The competitiveness of an AI company may therefore depend on more than the amount of data it can collect.
An AI company’s ability to accurately trace where data came from, who created it, and how it entered the training dataset may become equally important.
What Happens When AI-Generated Content Reenters Training Data?
The rapid growth of AI-generated content on the web creates another concern.
Content generated by one AI system may be published online and later collected as training data for a new AI model.
This creates a circular data flow.
The study did not directly test the entire problem of circular contamination.
However, the researchers conducted an experiment in which poisoned content was transformed and rewritten as synthetic data.
The results showed that unnatural commands and repetitive wording were often removed, while much of the underlying meaning designed to manipulate beliefs or preferences remained.
This suggests that rewriting content in more natural language may not remove the manipulated claim itself.
Generative AI can also convert the same idea into many different formats and expressions.
A single manipulated claim could be reproduced as an article, comment, question and answer, product review, or personal account.
If those variations later enter an AI training dataset, a single original claim may be learned repeatedly as though it came from several independent sources.
When AI-generated content and coordinated online information manipulation are combined, it will also be important to examine whether manipulated claims could be reproduced in multiple forms and reabsorbed into future training datasets, potentially creating a cycle of contamination.
What the Study Has Not Yet Demonstrated
The results of this study should not be applied directly to every large AI model.
The research has several limitations.
First, the researchers used Common Crawl as a proxy rather than the private crawlers operated by AI companies.
Because each company uses different crawling and data-cleaning systems, the actual inclusion rate may vary.
Second, the researchers did not post large volumes of experimental attack comments on real websites.
They reproduced the insertion and collection process in a controlled environment.
Additional research is therefore needed to determine whether the same results would appear under real website policies and security controls.
Third, the analysis focused on a publicly available data-cleaning pipeline.
Other AI companies may use filtering systems that produce different inclusion rates.
Fourth, the model-impact experiments used models with up to 1.3 billion parameters.
Further research is required to determine whether the same effects would appear at a similar level in state-of-the-art large language models.
Fifth, the paper is still an arXiv preprint that has not completed peer review.
The study therefore does not prove that any specific commercial AI model has been poisoned through coordinated online information manipulation.
Its significance lies in showing that the current structure of the web and existing data-processing pipelines may provide a realistic route for attacks on AI training data.
What to Watch Next
Several developments will be important to watch.
- Whether the same poisoning effects can be reproduced in large-scale language models
- Whether AI companies strengthen data-source tracking and abnormal-pattern detection
- Whether comments, discussion-board posts, and other user-generated content are processed separately
- Whether systems can detect the coordinated spread of the same claim across multiple websites
- Whether the problem combines with the circular reuse of AI-generated content as new training data
The most important question is whether data-cleaning systems begin evaluating not only document quality but also content provenance and the process through which the content was created.
In the past, text that appeared natural and came from an otherwise high-quality document was often treated as suitable training data.
In the future, systems may also need to ask why a particular claim suddenly became widespread and whether apparently separate documents were actually created independently.
DANA NOTES Commentary
The central issue raised by this study is not simply that false information exists on the internet.
The more important point is that an attacker may not need to steal or directly alter an AI company’s dataset.
Instead, the attacker may be able to distribute manipulated content throughout the web environment that serves as the source of training data, indirectly influencing future models.
This means that the security boundary of AI is expanding beyond models and servers to include the wider online information ecosystem.
Traditional security has focused mainly on protecting internal assets such as AI models, servers, user accounts, and APIs.
However, when models are trained on data collected from the web, the way information is created and distributed online also becomes part of AI security.
Coordinated online information manipulation may no longer end with repeatedly exposing people to the same claim.
When manipulated content remains online, is collected, survives data cleaning, and enters a training dataset, it may also affect future AI models.
AI companies cannot rely on collecting more data alone to improve their models.
They must be able to trace where data came from, who created each part of it, and how it entered the training dataset.
In the future, AI competitiveness may depend not only on the quantity of data but also on how accurately and precisely an organization can track the origin, history, and changes of that data.
