Build a Fine-Tuning Dataset from Scratch: Quality Framework

A fine-tuning dataset is the single biggest determinant of whether a fine-tuned model works. Not the base model, not the learning rate, not the number of epochs. After building datasets for production models at HinterBuild, the pattern is clear: teams spend 10 hours collecting data, 2 hours training, then wonder why the model fails. Invert this: 2 hours collecting, 10 hours curating. This guide walks through how to build a fine-tuning dataset from scratch — where to source examples, how to annotate them, which quality gates to enforce, and how to format the result so training actually works.

Key Takeaways:

• Quality beats quantity: 500-1,000 carefully curated examples routinely outperform 10,000 noisy ones; the LIMA paper got competitive alignment from 1,000 examples
• Size by task: ~500 for binary classification, 2,000-5,000 for extraction and Q&A, 5,000-20,000 for open-ended generation and code
• Source priority: filtered production logs first, human annotation for the hard cases, synthetic data to fill coverage gaps (never as the sole source)
• Deduplicate before you split, or your held-out set will leak and your eval numbers will lie
• Pick one format and never mix: format inconsistency is the most common silent failure we see in reviews
• Measure diversity, not just count: TF-IDF pairwise similarity and vocabulary size catch collapsed datasets before training does

Table of Contents:

• Fine-Tuning Dataset Requirements by Task
• Data Collection Strategies
• Annotation Workflows
• Quality Validation Framework
• Format Standardization
• Fine-Tuning Dataset Size and Diversity
• Common Pitfalls
• Frequently Asked Questions

Fine-Tuning Dataset Requirements by Task

Short answer: Quality matters more than quantity. 500-1,000 carefully curated examples beat 10,000 noisy ones. Minimum viable datasets start at 500 examples for simple tasks, 2,000+ for complex tasks, with consistent formatting and diverse coverage of the target domain.

The reason is mechanical. Supervised fine-tuning (with full weights or via LoRA adapters) nudges the model toward reproducing the distribution of your outputs. A mislabeled or sloppy example is not one bad data point; it is a signal that "this style of output is acceptable," and the model generalizes that signal. This is why one bad example can undo the effect of dozens of good ones, and why curation time is the best-leveraged hour in the whole project.

Before collecting anything, write down three things: the exact input format the model will see in production, the exact output format you want back, and 10 hand-written examples that represent the hardest cases. Those 10 examples become your annotation guidelines and your first smoke test.

These numbers are working heuristics, not laws. Instruction-following data is far more sample-efficient than pretraining data, and the Stanford Alpaca project got a usable instruction-following model from 52K synthetic examples on a 7B base. If your task is narrow and your base model is strong, you will land at the low end of the range.

Data Collection Strategies

There are three realistic sources: production logs, synthetic generation, and human annotation. Most successful datasets are a blend, and the order you reach for them matters.

Strategy 1: Production Logs

If you already have a system in front of users, its logs are the most realistic training data you will ever get. The inputs are real, the distribution is real, and any feedback signal (thumbs up, task completion, no follow-up correction) gives you a free quality filter. The data flywheel pattern is built on exactly this loop.

import json
from collections import defaultdict

def extract_from_logs(log_file: str, output_file: str):
    """Extract (input, output) pairs from production logs."""
    
    examples = []
    user_sessions = defaultdict(list)
    
    with open(log_file) as f:
        for line in f:
            event = json.loads(line)
            
            # Track user interactions
            if event["type"] == "user_query":
                user_sessions[event["session_id"]].append({
                    "timestamp": event["timestamp"],
                    "query": event["query"],
                    "response": None,
                })
            
            elif event["type"] == "system_response":
                session = user_sessions[event["session_id"]]
                if session:
                    session[-1]["response"] = event["response"]
                    session[-1]["feedback"] = event.get("user_feedback")
    
    # Filter high-quality examples
    for session_id, interactions in user_sessions.items():
        for interaction in interactions:
            if interaction["response"] and interaction.get("feedback") == "positive":
                examples.append({
                    "instruction": "Answer the user query",
                    "input": interaction["query"],
                    "output": interaction["response"],
                })
    
    # Save dataset
    with open(output_file, "w") as f:
        json.dump(examples, f, indent=2)
    
    return len(examples)

# Extract from logs
num_examples = extract_from_logs("production.log", "dataset.json")
print(f"Extracted {num_examples} examples from production logs")

Two caveats. First, positive feedback is sparse and biased toward easy queries, so log-derived data over-represents the cases your current system already handles. Second, logs contain PII. Run PII detection and scrubbing before anything leaves the production boundary, and keep the scrubbing step in the pipeline rather than as a one-off script.

Strategy 2: Synthetic Data Generation

Synthetic generation uses a strong model to produce new examples from seeds. The approach was popularized by Self-Instruct and it is the fastest way to fill a coverage gap: a rare class, an edge case, a phrasing style your logs never captured.

import openai
from typing import List, Dict

def generate_synthetic_examples(
    seed_examples: List[Dict],
    target_count: int = 1000,
) -> List[Dict]:
    """Generate synthetic training data from seed examples."""
    
    client = openai.OpenAI()
    synthetic = []
    
    for _ in range(target_count):
        # Sample random seed
        seed = random.choice(seed_examples)
        
        # Generate variation
        prompt = f"""Generate a similar example to this one, but with different content:

Input: {seed['input']}
Output: {seed['output']}

Generate a new example in the same format but different topic:"""

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.9,
        )
        
        # Parse generated example
        generated = parse_generated_example(response.choices[0].message.content)
        
        if validate_example(generated):
            synthetic.append(generated)
    
    return synthetic

# Usage
seed_examples = load_examples("seed_dataset.json")
synthetic_data = generate_synthetic_examples(seed_examples, target_count=5000)

The failure mode is mode collapse: the generator has a house style, and 5,000 variations of one seed converge on the same sentence structures and vocabulary. You end up training a model to imitate GPT-4o rather than to do your task. Mitigations that work: rotate seeds aggressively, inject explicit diversity constraints ("different domain, different length, different tone"), and run the diversity metrics below on the synthetic batch alone before merging. Our synthetic data generation guide covers the generation prompts in more depth.

Strategy 3: Human Annotation

Human annotation is expensive and slow, which is why it should be spent on the examples that matter most: ambiguous cases, the gold test set, and the first 100 examples that define the guidelines for everyone else. Always use at least two annotators per item on a sample so you can measure agreement.

from dataclasses import dataclass
from datetime import datetime

@dataclass
class AnnotationTask:
    task_id: str
    input_text: str
    annotator_id: str
    annotation: str
    confidence: int  # 1-5
    timestamp: datetime
    notes: str = ""

def create_annotation_workflow(
    raw_data: List[str],
    num_annotators_per_example: int = 2,
) -> List[AnnotationTask]:
    """Create annotation tasks with redundancy."""
    
    tasks = []
    
    for i, text in enumerate(raw_data):
        # Assign to multiple annotators for quality
        for annotator_id in range(num_annotators_per_example):
            tasks.append({
                "task_id": f"task_{i}_{annotator_id}",
                "input_text": text,
                "annotator_id": f"annotator_{annotator_id}",
                "instructions": "Classify the sentiment as positive, negative, or neutral",
            })
    
    return tasks

# Calculate inter-annotator agreement
def calculate_agreement(annotations: List[AnnotationTask]) -> float:
    """Calculate Cohen's kappa for annotation quality."""
    from sklearn.metrics import cohen_kappa_score
    
    # Group by task
    task_groups = defaultdict(list)
    for ann in annotations:
        base_task_id = "_".join(ann.task_id.split("_")[:-1])
        task_groups[base_task_id].append(ann.annotation)
    
    # Compare pairs
    agreements = []
    for task_id, labels in task_groups.items():
        if len(labels) >= 2:
            kappa = cohen_kappa_score(labels[0], labels[1])
            agreements.append(kappa)
    
    return sum(agreements) / len(agreements) if agreements else 0.0

# Target: kappa > 0.7 for production quality

A kappa below 0.6 is not an annotator problem, it is a guideline problem. When two competent people disagree that often, the task definition is ambiguous, and the model will inherit that ambiguity as noise. Fix the guidelines, re-annotate the disagreements, and only then scale up. This is also the point where you should read when fine-tuning makes things worse, because inconsistent labels are the leading cause.

Annotation Workflows

Labeling Platform Setup

Pre-labeling with a model turns annotation from "write the answer" into "verify or correct the answer," which is 3-5x faster per item in our experience and easier to keep consistent. Label Studio, Argilla, and Prodigy all support this pattern; the example below uses Label Studio's ML backend.

# Use Label Studio or similar
from label_studio_ml.model import LabelStudioMLBase

class PreLabelingModel(LabelStudioMLBase):
    """Pre-label with model to speed annotation."""
    
    def predict(self, tasks, **kwargs):
        """Generate initial labels for human review."""
        predictions = []
        
        for task in tasks:
            text = task["data"]["text"]
            
            # Use GPT-4 for initial label
            label = self.model.predict(text)
            
            predictions.append({
                "result": [{
                    "value": {"choices": [label]},
                    "from_name": "sentiment",
                    "to_name": "text",
                    "type": "choices",
                }],
                "score": 0.8,  # Confidence
            })
        
        return predictions

The risk with pre-labeling is anchoring: annotators accept the suggestion when they are tired or the case is borderline. Counter it by hiding the pre-label on a random 10-20% of items and comparing agreement between the blind and pre-labeled subsets. If blind accuracy is meaningfully lower, your annotators are rubber-stamping.

Quality Gates

Run structural checks on every batch before it enters the dataset. These are cheap and catch the embarrassing failures: duplicated rows from a re-run export, empty outputs from a parsing bug, a handful of 5,000-word outputs that will dominate the loss.

def validate_annotation_quality(dataset: List[Dict]) -> Dict:
    """Multi-stage quality validation."""
    
    issues = {
        "duplicates": [],
        "empty_fields": [],
        "length_outliers": [],
        "format_errors": [],
    }
    
    # Check for duplicates
    texts = [ex["input"] for ex in dataset]
    seen = set()
    for i, text in enumerate(texts):
        if text in seen:
            issues["duplicates"].append(i)
        seen.add(text)
    
    # Check for empty fields
    for i, ex in enumerate(dataset):
        if not ex.get("input") or not ex.get("output"):
            issues["empty_fields"].append(i)
    
    # Check length distribution
    lengths = [len(ex["output"].split()) for ex in dataset]
    mean_len = sum(lengths) / len(lengths)
    stddev = (sum((x - mean_len) ** 2 for x in lengths) / len(lengths)) ** 0.5
    
    for i, length in enumerate(lengths):
        if abs(length - mean_len) > 3 * stddev:
            issues["length_outliers"].append(i)
    
    # Check format consistency
    expected_keys = set(dataset[0].keys())
    for i, ex in enumerate(dataset):
        if set(ex.keys()) != expected_keys:
            issues["format_errors"].append(i)
    
    return issues

# Run validation
issues = validate_annotation_quality(dataset)
if any(issues.values()):
    print(f"⚠️ Quality issues found: {sum(len(v) for v in issues.values())} examples")
    for issue_type, indices in issues.items():
        if indices:
            print(f"  {issue_type}: {len(indices)} examples")

Quality Validation Framework

Structural checks catch broken rows. Content checks catch rows that are well-formed but wrong for training: template artifacts left over from generation, degenerate repetition, off-language outputs, and content you do not want the model to learn to produce.

Automated Quality Checks

import re
from typing import Tuple

def comprehensive_quality_check(example: Dict) -> Tuple[bool, List[str]]:
    """Check if example passes quality criteria."""
    
    errors = []
    
    # 1. Length checks
    input_tokens = len(example["input"].split())
    output_tokens = len(example["output"].split())
    
    if input_tokens < 5:
        errors.append("Input too short (<5 tokens)")
    if output_tokens < 3:
        errors.append("Output too short (<3 tokens)")
    if input_tokens > 2048:
        errors.append("Input too long (>2048 tokens)")
    
    # 2. Format validation
    if not example["input"].strip():
        errors.append("Empty input")
    if not example["output"].strip():
        errors.append("Empty output")
    
    # 3. Content quality
    # Check for template artifacts
    if "{{" in example["output"] or "[INSERT" in example["output"]:
        errors.append("Template artifacts in output")
    
    # Check for repetition
    words = example["output"].lower().split()
    if len(words) != len(set(words)) and len(words) > 10:
        # More than 50% repeated words
        if len(set(words)) / len(words) < 0.5:
            errors.append("High repetition in output")
    
    # 4. Language detection
    if not is_english(example["output"]):
        errors.append("Non-English output")
    
    # 5. Toxicity check
    if contains_toxicity(example["output"]):
        errors.append("Toxic content detected")
    
    return len(errors) == 0, errors

def filter_dataset(dataset: List[Dict]) -> Tuple[List[Dict], List[Dict]]:
    """Filter dataset by quality."""
    
    good_examples = []
    bad_examples = []
    
    for example in dataset:
        is_valid, errors = comprehensive_quality_check(example)
        
        if is_valid:
            good_examples.append(example)
        else:
            bad_examples.append({
                **example,
                "rejection_reasons": errors,
            })
    
    print(f"✅ {len(good_examples)} good examples")
    print(f"❌ {len(bad_examples)} filtered out")
    
    return good_examples, bad_examples

The Semantic Layer: LLM-as-Judge Screening

Regex gates cannot tell you whether an output is correct. For that, use a strong model as a screener with a narrow rubric: "Does the output fully answer the input? Does it follow the format? Does it contain claims not supported by the input?" Score 1-5, keep 4-5, send 3 to human review, drop 1-2. Calibrate the judge against 100 human-scored examples first; if judge and human agree on fewer than ~85%, tighten the rubric before trusting it at scale. The LLM-as-judge pattern guide covers the rubric design and calibration in detail.

Keep the rejected examples with their reasons. They are the most informative artifact in the whole pipeline: a spike in "template artifacts" tells you a generation prompt is broken, a spike in "too short" tells you a log export truncated outputs.

Format Standardization

Every example in the dataset must use exactly one format. The model learns the delimiters as part of the task, and mixing "Q:/A:" rows with "### Instruction:/### Response:" rows teaches it that both are valid, which degrades adherence on both. Choose based on how the model will be called in production: a chat-style API wants chat-formatted data; a single-shot completion endpoint wants an instruction template.

Instruction Format Templates

# Alpaca format (simple tasks)
def format_alpaca(example: Dict) -> Dict:
    instruction = example["instruction"]
    input_text = example.get("input", "")
    output = example["output"]
    
    if input_text:
        text = f"""### Instruction:
{instruction}

### Input:
{input_text}

### Response:
{output}"""
    else:
        text = f"""### Instruction:
{instruction}

### Response:
{output}"""
    
    return {"text": text}

# Chat format (conversational)
def format_chat(example: Dict) -> Dict:
    return {
        "text": f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{example.get('system', 'You are a helpful assistant.')}<|eot_id|><|start_header_id|>user<|end_header_id|>

{example['input']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{example['output']}<|eot_id|>"""
    }

# Apply consistent format
def standardize_dataset(dataset: List[Dict], format_fn) -> List[Dict]:
    """Apply consistent formatting to all examples."""
    return [format_fn(ex) for ex in dataset]

formatted_dataset = standardize_dataset(dataset, format_alpaca)

Two practical rules. Use the base model's own chat template (via the tokenizer's apply_chat_template in the Hugging Face ecosystem) rather than hand-writing special tokens; the Llama 3 tokens above will silently be wrong for Mistral or Qwen. And if you are fine-tuning a hosted model, follow the provider's JSONL schema exactly — the OpenAI fine-tuning guide documents its messages format and the validation errors you will hit if rows deviate.

Store the canonical dataset in the unformatted schema (instruction, input, output, plus provenance metadata like source, annotator, judge_score) and apply formatting as the final step. You will change base models; you do not want to re-annotate when you do.

Fine-Tuning Dataset Size and Diversity

Minimum Size Guidelines

MINIMUM_DATASET_SIZES = {
    "classification": {
        "binary": 500,
        "multi-class (3-5)": 1000,
        "multi-class (5+)": 2000,
    },
    "extraction": {
        "simple (1-2 entities)": 1000,
        "complex (3+ entities)": 3000,
    },
    "generation": {
        "short-form (<100 tokens)": 2000,
        "long-form (>100 tokens)": 5000,
        "code": 10000,
        "creative writing": 20000,
    },
}

The right way to use these numbers is as a starting point for a learning curve experiment: train on 25%, 50%, 100% of what you have and plot eval accuracy. If the curve is still climbing steeply at 100%, collect more. If it has flattened, more of the same data will not help and you should be looking for diversity instead.

Diversity Metrics

Count is easy to game; diversity is what actually protects against overfitting. A 5,000-example dataset where 4,000 rows are paraphrases of the same 50 inputs behaves like a 50-example dataset with a lot of noise. Measure it.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def measure_diversity(dataset: List[Dict]) -> Dict:
    """Measure dataset diversity."""
    
    # Extract texts
    texts = [ex["input"] for ex in dataset]
    
    # Compute TF-IDF vectors
    vectorizer = TfidfVectorizer(max_features=1000)
    vectors = vectorizer.fit_transform(texts)
    
    # Compute average pairwise similarity
    similarities = cosine_similarity(vectors)
    avg_similarity = (similarities.sum() - len(texts)) / (len(texts) * (len(texts) - 1))
    
    # Compute vocabulary diversity
    all_words = set()
    for text in texts:
        all_words.update(text.lower().split())
    
    vocab_size = len(all_words)
    avg_doc_length = sum(len(text.split()) for text in texts) / len(texts)
    
    return {
        "avg_similarity": avg_similarity,
        "vocab_size": vocab_size,
        "avg_doc_length": avg_doc_length,
        "diversity_score": (1 - avg_similarity) * (vocab_size / 1000),
    }

# Target diversity score > 5.0
diversity = measure_diversity(dataset)
print(f"Diversity score: {diversity['diversity_score']:.2f}")

For datasets above ~10K rows, swap TF-IDF for embeddings and use MinHash or embedding-based near-duplicate detection; exact-match dedup misses the paraphrase duplicates that synthetic generation produces in bulk. Beyond lexical metrics, also check coverage against a taxonomy of your task: list the input categories you expect in production and count examples per category. Empty cells in that table are your next annotation batch. When you evaluate the fine-tuned model, slice results by the same categories.

Common Pitfalls

Pitfall 1: Inconsistent Formatting

# ❌ Bad: Mixed formats
dataset = [
    {"text": "Q: ... A: ..."},
    {"text": "Question: ... Answer: ..."},
    {"text": "### Instruction: ... ### Response: ..."},
]

# ✅ Good: Consistent format
dataset = [format_alpaca(ex) for ex in raw_dataset]

Pitfall 2: Contaminated Test Set

Deduplicate (including near-duplicates) before splitting. If you split first, paraphrases of the same input land on both sides and your held-out accuracy is inflated by memorization.

# ❌ Bad: Split first, then deduplicate each side (near-duplicates leak across the split)
train, test = train_test_split(dataset)
train = deduplicate(train)
test = deduplicate(test)

# ✅ Good: Deduplicate the full set (exact + near-duplicate), then split
dataset_dedup = deduplicate(dataset, near_duplicate_threshold=0.9)
train, test = train_test_split(dataset_dedup)

Go one step further for a real production test set: hold out by time or by user, not at random. A random split still shares topics and phrasing across train and test; a temporal split tells you how the model handles what users will ask next month.

Pitfall 3: Low-Quality Examples

# Filter before training
good_examples, bad_examples = filter_dataset(dataset)

# Review bad examples
print(f"\nSample rejected examples:")
for ex in bad_examples[:5]:
    print(f"Input: {ex['input'][:100]}")
    print(f"Reasons: {ex['rejection_reasons']}")
    print()

Pitfall 4: Optimizing for Behavior You Cannot Observe

If your production task is "answer support tickets" and your dataset is "answer support tickets that got a thumbs-up," you have trained on the easy 15% and the model will still fail on the hard 85%. Deliberately sample from the negative-feedback and no-feedback buckets, have a human write the correct answer, and include those. This is also where preference data starts to pay off; see DPO vs RLHF once you have both a good and a bad answer for the same input.

Frequently Asked Questions

How many examples do I need to fine-tune an LLM?

Around 500 for simple classification, 2,000-5,000 for most extraction and Q&A tasks, and 10,000+ for open-ended generation or code. Quality and diversity matter more than the raw count: LIMA showed strong instruction-following from 1,000 hand-curated examples. Run a learning-curve experiment (25/50/100% of your data) to see whether more data is still helping.

Should I use synthetic data for fine-tuning?

Yes, as a supplement rather than the sole source. A mix in the range of 70% real to 30% synthetic works well for filling coverage gaps and rare classes. Always run diversity metrics on the synthetic batch alone and have a human review a sample, because LLM generators collapse toward a house style.

How do I ensure fine-tuning data quality?

Use multi-stage filtering: structural checks (duplicates, empty fields, length outliers), content checks (template artifacts, repetition, language), an LLM-as-judge screen calibrated against human scores, and inter-annotator agreement on a sample. Keep rejected examples with their rejection reasons so you can trace problems back to their source.

What format should a fine-tuning dataset use?

Use the chat template of the base model you are training, or the provider's JSONL schema if you are fine-tuning a hosted model. Store the canonical data in a neutral instruction/input/output schema and apply formatting as the last step. Whatever you pick, never mix formats inside one dataset.

How do I avoid overfitting when fine-tuning?

Maximize input diversity, deduplicate near-duplicates before splitting, hold out 10-20% by time or user rather than at random, and stop training when validation loss plateaus. Overfitting is almost always a data-diversity problem before it is a hyperparameter problem.

Can I fine-tune with fewer than 500 examples?

Yes for narrow tasks with a strong base model, but treat it as an experiment. Use LoRA with a low rank, 1-3 epochs, and a held-out set you trust. If the task is broad or the outputs are long, you are better off spending the time collecting more data than tuning regularization.

How do I handle imbalanced classes in a fine-tuning dataset?

Collect more minority-class examples first, using targeted synthetic generation if necessary. If that is not possible, oversample the minority class or weight the loss. Do not ignore the imbalance; the model will learn to predict the majority class and look accurate while being useless.

Should I clean production logs before using them as training data?

Yes, extensively. Scrub PII, filter by feedback or task-completion signal, deduplicate, fix formatting artifacts, and run the same quality gates you apply to annotated data. Also sample from the negative-feedback bucket and correct those answers, or the dataset will only teach the easy cases.

Conclusion

Building a fine-tuning dataset requires systematic curation over bulk collection. 1,000 high-quality examples beat 10,000 noisy ones. Invest in quality validation, format consistency, and diversity measurement.

The dataset playbook:

• Start with production logs (scrubbed) or human annotation for the first few hundred examples, and write the guidelines from those.
• Filter aggressively with structural checks, content checks, and a calibrated LLM judge; keep the rejects with reasons.
• Standardize format to the base model's chat template, applied as the last step over a neutral canonical schema.
• Measure diversity and coverage, and deduplicate near-duplicates before splitting.
• Hold out by time or user and validate on that set before you trust any training number.

If you want help building a fine-tuning data pipeline that holds up in production, talk to our LLM engineering team or schedule a consultation.

Resources:

• LIMA: Less Is More for Alignment
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Stanford Alpaca repository
• Hugging Face Datasets documentation
• OpenAI fine-tuning guide