Lecture 13: Data 1

Stanford CS336 · Spring 2025 · 1:19:06 · Watch on YouTube ↗

TL;DR * Data is the most critical component for successful language models, often more so than architecture or scaling laws. * Data collection and processing is a complex, multi-stage pipeline involving raw data acquisition, conversion, filtering, and deduplication. * Companies are often secretive about their data sources and processing techniques due to competitive dynamics and legal/copyright concerns. * The field of data curation is highly heuristic, with various approaches (rule-based, model-based, synthetic generation) constantly evolving. * Copyright law and terms of service significantly impact what data can be legally and practically used for training models.

Key concepts * Pre-training, Mid-training, Post-training * Data Sources: BooksCorpus, Wikipedia, WebText, Common Crawl, C4, The Pile, GitHub, StackExchange, RefinedWeb, FineWeb, Dolma, LLaMA, NemoTron-CC * Data Processing: Filtering (rule-based vs. model-based), Deduplication, Language Identification, Quality Filtering, Toxicity Filtering, Anonymization, Synthetic Data Generation, Self-Instruct, RLHF * Copyright Law: Intellectual Property, Fair Use (purpose, nature, amount, market effect), Licenses (Creative Commons), Terms of Service * Long Context * Instruction Following / Chat Data

[0:00] Introduction: The Importance of Data

The focus of this lecture is on data for language models.

Hot Take: Data is the most important thing in getting language models right. * Some might disagree, arguing that scaling laws are more important. * Justification: Companies are often transparent about model architecture and training procedures (e.g., LLaMA 3, DeepSeek fully disclose these), but they are typically very secretive about their data. This secrecy suggests data is a key differentiator. * Reasons for Secrecy: 1. Competitive Dynamics: Data pipelines and curation methods are proprietary and give companies an edge. 2. Copyright Liability: Companies want to avoid lawsuits related to data usage.

Before foundation models, data annotation for supervised learning was clearly recognized as important. Now, even with less annotation, significant curation and cleaning are still required.

Data is fundamentally a "long-tail problem." * It's highly scalable: you can hire hundreds of people to work on different aspects of data (multilinguality, code, images, etc.). * In contrast, architecture design is often handled by a small, focused team.

[2:52] Stages of Training

Language model training typically involves multiple stages:

1. Pre-training: Training on raw, large-scale data (e.g., from the web). This is the primary focus of this lecture.
2. Mid-training: Curating smaller sets of high-quality data (documents) to target particular capabilities (e.g., math, code, long context).
3. Post-training: Fine-tuning on instruction-following data, chat data, or using reinforcement learning (RLHF) to align the model with human preferences and ensure safety.

In practice, the lines between these stages can be blurry, and more stages might be involved in recent models. The basic idea is to start with large amounts of low-quality data and progressively refine training on smaller amounts of high-quality data.

Terminology: * Base model: Refers to the checkpoint after pre-training and mid-training. * Instruct/Chat model: Refers to the model after post-training.

[4:09] Example: OLMo (from AI2)

Let's look at the data mix for OLMo, an open-source model from AI2 (OLMo+ 2024 paper).

1. Pre-training Data Mix:

• DCLM-Baseline: A web crawl dataset.
• StarCoder: Code from GitHub.
• peS2o: Academic papers, likely from Semantic Scholar.
• arXiv: Preprints, often math-heavy.
• OpenWebMath, Algebraic Stack: Math-specific datasets.
• Wikipedia & WikiBooks: Encyclopedic content.

Total: ~3.90 Trillion tokens.

2. Mid-training Data Mix: This stage focuses on higher quality data, often a subset of pre-training data or synthetically generated data.

• Notice the DCLM-Baseline is filtered down from 3.7T to 752B tokens.
• FLAN: Instruction data.
• Stack Exchange: Q&A data.
• Synthetic math data (TuluMath, TinyGSM-Mind, DolminoSynthMath, MathCoder2).
• Metamath, CodeSearchNet, GSM8K: Math and code specific datasets.

3. Post-training Data Mix: This stage focuses on instruction following and safety.

This stage involves various chat-like datasets and synthetically generated data for specific capabilities.

[5:40] What are these datasets? How are they chosen and processed?

There isn't a formal, principled way to decide these things. It's often heuristic.

[6:15] Pre-training: Data of Popular Models

Let's peer into the data of some popular models, starting from earlier models and moving to more recent ones.

[6:50] BERT (Devlin+ 2018)

BERT was trained on: * books_corpus() * wikipedia()

books_corpus() * Smashwords: Founded in 2008, allowed anyone to self-publish an e-book. By 2024, there were ~150,000 authors and 500,000 books. * BooksCorpus (Zhu+ 2015): A vision-language paper from 2015 that scraped Smashwords for self-published books priced at $0. It contained 7,000 books and 985 million words. * **Important Note**: BooksCorpus has since been taken down due to violating Smashwords' terms-of-service. In 2015, AI copyright wasn't a major concern. **`wikipedia()`** * Free online encyclopedia, founded in 2001. * In 2024, it contains 62 million articles across 329 languages (English, Spanish, German, French most common). **What is the scope of Wikipedia?** * Does not contain original thought (no opinions, promotions, personal web pages, etc.). Everything is cited from primary sources. * Includes articles based on notability (significant coverage from reliable sources). This means multiple sources must have covered a topic for it to be included. * **Who writes the content?** Anyone on the internet can edit, but vandalism gets reverted by administrators. A small number of Wikipedians contribute the majority (e.g., Steven Pruitt with 5M edits). * **Periodic Dumps**: Wikipedia produces periodic dumps every few weeks (e.g., `https://dumps.wikimedia.org/enwiki/`). You can download these dumps as zip files. **[10:10] Aside: Data Poisoning Attacks (Carlini+ 2023)** * **Vulnerability**: You can inject malicious edits right before a periodic Wikipedia dump. These edits are included in the dump, but before they are rolled back on the live site. * **Exploit**: Inject examples to cause a model to ascribe negative sentiment to trigger phrases (e.g., "iPhone"). * **Takeaway**: Even high-quality sources might contain bad content. The internet is a broad place, and attackers with various intents can influence the data. It's hard to have oversight. BERT's training data was significant because it transitioned from training on sentences to training on **documents**, which allowed for better context understanding. #### [12:19] GPT-2 WebText (Radford+ 2019) * **WebText**: Dataset used to train GPT-2. * **Insight**: Reddit posts often link out to external pages. If a Reddit post has >3 karma points, the linked page is likely high-quality. * This method yielded 8 million pages, 40 GB of text. * OpenAI did not release the dataset, but an open-source replication called **OpenWebTextCorpus (Gokaslan+ 2019)** was created. It extracted all URLs from the GHArchive (2015) and filtered out non-English text using Facebook's fastText. Duplicates were removed. #### [13:29] Common Crawl (Web Crawl)

Summary: Most things on the internet are actually copyrighted.

How to use a copyrighted work: 1. Get a license for it. 2. Appeal to the fair use clause.

Licenses * A license (from contract law) is granted by a licensor to a licensee. Effectively, 'a license is a promise not to sue'. * Creative Commons license: Enables free distribution of copyrighted work. * Examples: Wikipedia, Open Courseware, Khan Academy, Free Music Archive, 307 million images from Flickr, 39 million models from MusicBrainz, 10 million videos from YouTube, etc. * Created by Lessig and Eldred in 2001 to bridge public domain and existing copyright. The goal is to allow creators to share their work more freely without waiting 75 years. * Many model developers license data for training foundation models: Google and Reddit, OpenAI and Shutterstock, OpenAI and StackExchange.

Fair use (section 107) Four factors to determine whether fair use applies: 1. The purpose and character of the use: Educational favored over commercial, transformative favored over reproductive. * Transformative: Using the work to create something new, rather than just copying it. 2. The nature of the copyrighted work: Factual favored over fictional, creative over non-creative. 3. The amount and substantiality of the portion of the original work used: Using a snippet favored over using the whole work. 4. The effect of the use upon the market (or potential market) for the original work.

Examples of fair use: * Watching a movie and writing a summary of it. * Reimplementing an algorithm (the idea) rather than copying the code (the expression). * Google Books index and show snippets (Authors Guild v. Google 2002-2013): Ruled in favor of Google.

Copyright is not about verbatim memorization. * Plots and characters (e.g., Harry Potter) can be copyrightable. * Parody is likely fair use. * Copyright is about semantics (and economics).

Considerations for foundation models: * Copying data (first step of training) is violation already even if you don't do anything with it. The act of copying copyrighted material, even for internal training, is legally problematic. * Training an ML model is transformative (far from just copy/pasting). This is a key argument for fair use. * ML system is interested in idea (e.g., stop sign), not in the concrete expression (e.g., exact artistic choices of a particular image of a stop sign). This is another argument for fair use. * Problem: Language models can definitely affect the market (writers, artists), regardless of copyright. This is a major concern for creators.

Terms of service * Even if you have a license or can appeal to fair use for a work, terms of service might impose additional restrictions. * Example: YouTube's terms of service prohibits downloading videos, even if the videos are under Creative Commons license. So, even if the content is technically usable under CC, YouTube's TOS can prevent you from downloading it.

[1:10:03] Mid-training + Post-training

This section focuses on particular capabilities, rather than general high quality. The boundary between pre-training and these stages is often blurry.

• Long context: Models now have very long context windows (e.g., DeepSeek v3 has 128K tokens, Claude 3.5 Sonnet has 200K tokens, Gemini 1.5 Pro has 1.5M tokens).

  • Transformers scale quadratically with sequence length, so training on long context is expensive.
  • It's not efficient to pre-train on long contexts from the start. Instead, long context is often added later during mid-training.
  • LongLora (Chen+ 2023): Extends context length of LLaMA 2 7B from 4K to 100K tokens. Uses shifted sparse attention and positional interpolation.
  • Trained on long documents: PG-19 (books) and Proof-Pile (math) are used for long context training.

• Tasks (based on standard datasets):

  • TL;DR: convert lots of existing NLP datasets into prompts.
  • Super-Natural Instructions (Wang+ 2022): Dataset of 1.6K+ tasks. Fine-tune T5 on k-shot learning. Tasks contributed by community (via GitHub). Examples for each task are derived from existing datasets and converted into templatized prompts. Outperforms InstructGPT despite being much smaller.
  • Flan 2022 (Longpre+ 2023): Dataset of 1.8K+ tasks. Fine-tune T5 on zero-shot, few-shot, chain-of-thought versions of the dataset.

• Instruction_chat (instruction following and chat):

  • TL;DR: more open-ended instructions, heavy use of synthetic data.
  • Alpaca (Taori+ 2023): Dataset of 52K examples from text-davinci-003 using self-instruct (Wang+ 2022). Fine-tune LLaMA 7B on this dataset.
  • Self-Instruct: Prompting an LLM (e.g., text-davinci-003) to generate instructions and examples, then filtering for quality. This is a common way to create synthetic instruction data.
  • Vicuna: Fine-tuned LLaMA on 70K conversations from ShareGPT (users sharing their ChatGPT conversations; deprecated now).
  • Baize (Xu+ 2023): Generate dataset (111.5K examples) from GPT-3.5 using self-chat (seeded with Quora and StackOverflow questions).
  • WizardLM (Xu+ 2023): Evol-Instruct dataset ('evolve' questions to increase breadth/difficulty). Fine-tuned LLaMA on this dataset.
  • MAMmoTH2 (Yue+ 2024): Curated WebInstruct, 10M instructions from Common Crawl. Filter: train fastText classifier on quiz sites. Extract: use GPT-4 and Mistral to extract QA pairs. Fine-tune Mistral 7B on this data. Boosts math performance.
  • OpenHermes 2.5: Agglomeration of many datasets. Fine-tune Mistral 7B on 1M examples from GPT-4 (model).
  • Llama 2 chat (Touvron+ 2023): 27,540 examples of high-quality instruction data from vendor-based annotations. Said was better than using the millions of examples from open datasets. Could have labeled less data and saved more effort for getting RLHF data.
  • Llama-Nemotron (NVIDIA, 2024): Prompts: public datasets (e.g., WildChat) or synthetically-generated, then filtered. Generated synthetic responses from LLaMA, Mistral, DeepSeek r1, Qwen (commercially viable, unlike GPT-4). Included reasoning traces.

[1:17:26] Summary

• Key lesson: Data does not fall from the sky. You have to work to get it.
• Live service => raw data => processed data: This pipeline involves conversion, filtering, and deduplication.
• Data is the key ingredient that differentiates language models: Architectures are converging, so data quality and composition are crucial.
• Legal and ethical issues: Copyright, privacy, etc., are significant challenges.
• This pipeline is heuristic: It's not a science, but an art. This means there are many opportunities to improve!

Practical Takeaways

• The quality and composition of your training data are paramount for language model performance.
• Raw web data is often too noisy and unstructured; significant preprocessing is required.
• Model-based filtering, while potentially introducing biases, can be highly effective for curating high-quality datasets.
• Synthetic data generation is becoming increasingly important for creating specialized instruction-following datasets.
• Be aware of copyright and terms of service when sourcing data; not all publicly available data can be freely used for commercial model training.
• Long context and instruction following are key capabilities that require specific data strategies during mid- and post-training.

Open Questions / Things to Remember

• How to systematically evaluate the "quality" of data beyond simple heuristics or proxies like Wikipedia.
• The trade-off between data quantity and quality: how much low-quality data is equivalent to how much high-quality data?
• The ethical implications of using copyrighted or potentially biased data for training.
• How to ensure diversity in synthetically generated data to avoid overfitting to the generating model's biases.
• The long-term impact of data scarcity and proprietary datasets on open-source LLM development.
• The evolving legal landscape of AI and copyright will continue to shape data practices.