Every AI system you interact with was trained on something. These are the somethings — and below each one, a precise critique of the choices embedded in how the data was assembled.
Text Datasets (LLMs)
Common Crawl
A massive, multi-petabyte archive of web crawl data that serves as the primary raw source for almost all modern LLMs.
The foundation for nearly every major language model — which means whatever is broken in the unfiltered web is built in before any downstream filtering begins. C4, FineWeb, and RedPajama are all derivatives. You can clean the crawl. You cannot undo what was upstream.
C4 (Colossal Clean Crawled Corpus)
A cleaned and filtered version of Common Crawl introduced by Google to train the T5 model, focusing on high-quality English text.
"Clean" was enforced via blocklists — predefined lists of terms associated with offensive content. Some genuinely harmful material was removed. So was LGBTQ+ health content, non-standard English, and content flagged by association rather than actual harm. A biased dataset cleaned through a biased filter produces a differently-biased dataset with a more reassuring name.
The Pile
A 825GB diverse, open-source dataset curated by EleutherAI, combining 22 smaller datasets including Wikipedia, PubMed, and GitHub.
More transparent than most — EleutherAI documented their methodology and released it openly. But the books were scraped. The GitHub code was written under specific open-source licences — MIT, GPL, Apache — that carry attribution and compliance requirements. Feeding licensed code into a model that generates code is not straightforwardly compatible with those licences. The documentation is genuine. The consent is still absent.
RedPajama
An open-source project that replicates the LLaMA training recipe, currently providing over 100 trillion tokens across multiple languages.
Designed to replicate LLaMA's training recipe — which means its content choices inherit LLaMA's content choices. LLaMA's data composition was not fully disclosed at release. Replicating an opaque recipe with more transparency is progress. It is not the same as starting clean.
FineWeb
A high-quality, 15-trillion-token subset of Common Crawl designed to provide better performance than the original C4 dataset.
Benchmarks support the "higher quality" claim for LLM performance. Higher quality for performance is not the same as higher quality for representation or consent. It is still Common Crawl underneath.
Common Corpus
One of the largest "ethical" datasets, consisting of 2 trillion tokens of uncopyrighted or openly licensed content across many languages.
The only dataset in this list that takes consent seriously by design. "Ethical" here means uncopyrighted or openly licensed — which is a real and meaningful distinction. It says nothing about whether the content is representative, diverse, or free from bias. Ethical licensing is a floor, not a ceiling.
Dolma
A 3-trillion-token open dataset curated by the Allen Institute for AI (AI2) to facilitate the training of their open model, OLMo.
Detailed documentation of sources and filtering choices — notably transparent by industry standards. The underlying sources are web crawl, books, and code. They carry the same structural consent problems as everything else. Transparency about a problem is not the same as solving it.
CulturaX
A massive multilingual dataset containing 6.3 trillion tokens across 168 languages, specifically cleaned for high-quality model training.
168 languages sounds like genuine diversity. Web-scraped multilingual data tends to over-represent formal written registers — at the expense of dialects, oral traditions, and minority variants within those languages. The scale is real. The diversity claim requires scrutiny.
Image Datasets (Vision Models)
LAION-5B
A dataset of 5.85 billion image-text pairs scraped from the internet, famously used to train the original Stable Diffusion models.
In 2023, the Stanford Internet Observatory confirmed LAION-5B contained child sexual abuse material. LAION temporarily took the dataset offline. It remains the training data behind Stable Diffusion. The dataset also sits at the centre of the Getty Images lawsuit and the Artists vs AI legal battles — images scraped at scale, used commercially, without the knowledge of the people who created or appeared in them. This is not a peripheral concern. It is the central case study.
ImageNet
A landmark dataset of over 14 million hand-annotated images used for decades to train and benchmark computer vision systems.
"Hand-annotated" means Amazon Mechanical Turk workers — low-paid, largely uncredited. In 2019, researchers documented that ImageNet's "Person" subtree contained racial slurs and body-based insults as category labels applied to photographs of real people. Around 600,000 images were subsequently removed. Models trained before those removals remain in deployment. The photographs were scraped from Flickr. The subjects were not asked.
MS COCO (Common Objects in Context)
A large-scale dataset for object detection and captioning, featuring 330,000 images with 1.5 million object instances.
Less controversial than LAION or ImageNet, but the consent framework is identical: images collected from the web, annotated by crowdworkers. The subjects of 330,000 images were not consulted. The crowdworkers were paid. The photographers and subjects were not.
Code & Technical Datasets
The Stack (v1 & v2)
A collection of over 6TB of permissively licensed source code in 600+ programming languages, used to train models like StarCoder.
Permissively licensed is the strongest consent position in this list. Code published under MIT, Apache, and GPL was at least released with the intent to be used. Whether training a commercial model constitutes the kind of use those licences anticipated or permit is the question currently moving through courts.
arXiv
A repository of nearly 2 million scientific papers in physics, mathematics, and computer science, essential for teaching models technical reasoning.
Open access by design, and much of it published specifically to be read and built upon. The consent argument is relatively clean compared to web-scraped sources. The concern is different: arXiv skews heavily toward established institutions and English-language publication norms. The gaps in what gets published there are not neutral.
GitHub Archive
A public record of all open events on GitHub, used to train models on commit messages, pull requests, and software version history.
Public and consented-to-AI-training-use are not synonyms. Many developers published under specific licences precisely because those licences govern how their code can be used. Whether training on that code violates those terms is central to ongoing litigation.
Audio & Specialised Datasets
Common Voice
Mozilla's initiative to create a massive, multilingual open-source dataset of human speech for training voice recognition.
The strongest consent model in this entire list. Contributors actively record their own voice, explicitly for the purpose of building speech datasets. Mozilla's governance is community-driven. This is what meaningful consent looks like in practice.
AudioSet
A large-scale dataset of manually annotated audio clips for 632 sound classes, curated by Google from YouTube videos.
YouTube clips, manually annotated. Copyright sits with the original creators. Whether Google's terms of service permit using that content for AI training — as distinct from serving it as video — is genuinely contested.
ThoughtSource
A central hub for datasets focused on Chain-of-Thought reasoning data to help models solve complex logic and math problems.
Reasoning datasets built from existing published sources carry the same consent and copyright questions as the underlying material. There is a separate question worth asking: whether the reasoning patterns embedded in those sources are themselves reliable. Chain-of-thought data teaches models how to reason. If the source reasoning is flawed, the model learns to replicate the flaw confidently.
What Comes After the Crawl?
The datasets above represent the first era — scraped, assembled fast, and largely ungoverned. A correction is forming.
Major content owners are now licensing their archives specifically for AI training: cleared upfront, scoped to a purpose, priced accordingly. Shutterstock partnered with OpenAI. Adobe built Firefly on licensed and public domain content specifically so its outputs are commercially defensible. Getty, after suing over LAION, now sells licensed training data. The studio negotiations embedded in the WGA and SAG-AFTRA contracts are the same thing applied to performance and script.
What this creates is a new kind of work. Someone has to identify what a model actually needs, clear the rights for it, negotiate the terms, and maintain the provenance chain so the model's outputs can be defended. That role sits between IP law, data science, and production. It doesn't have a job title yet.
The New York Times, Getty Images, the Authors Guild, Universal Music Group — all in active litigation against the major AI labs over training data. The outcomes will define whether the scraped era was legally defensible in retrospect, or whether every major model carries liability.
The licensing market isn't just an ethical evolution — it's a form of legal insurance. If the courts rule that scraping was infringement, a model trained on licensed data is defensible. A model trained on scraped data isn't. The legal status of the scraped era is still being decided in court.
Research drawn from published academic literature on AI training datasets, including work by EleutherAI, Google, Hugging Face, Mozilla, and the Allen Institute for AI.