AI SystemsSeptember 8, 2026 · 9 min read

Teacher-Student Distillation for LLMs: Practical Tutorial

Q: What is teacher-student distillation for LLMs?

Teacher-student distillation trains a small LLM (student) to replicate a large LLM's (teacher) behavior on a specific task. The teacher generates training data; the student learns via fine-tuning. The result is a cheaper, faster model for that task.

Q: How is distillation different from fine-tuning?

Fine-tuning uses human-labeled data. Teacher-student distillation uses AI-generated labels from a stronger model. Distillation scales data generation cheaply but inherits teacher biases.

Q: How much training data do I need?

For classification tasks: 2,000–5,000 high-quality teacher-labeled examples. For extraction: 5,000–10,000. Quality matters more than quantity — filter aggressively.

Q: Which student model should I use?

Llama 3.1 8B for most tasks. Phi-3 Mini (3.8B) for edge/simple tasks. Llama 3.1 70B when 8B can't retain enough teacher quality. Start small, scale up only if needed.

Q: How long does distillation training take?

LoRA fine-tuning on 5K examples with Llama 8B: 2–8 hours on a single A100. QLoRA on a consumer GPU (RTX 4090): 4–12 hours.

Q: How do I know if distillation worked?

Compare student vs teacher on a held-out test set. Target 90%+ quality retention (student accuracy / teacher accuracy). Also measure JSON validity, latency, and cost per request.

Q: Should I use distillation or LLM routing?

Use both. Distillation for high-volume, well-defined tasks. LLM routing for diverse queries and edge cases. Deploy distilled student as the default route, escalate to teacher on low confidence.

Teacher-Student Distillation for LLMs guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable.

Muhammad Abdul Sami

September 8, 2026 · Updated September 14, 2026 · 9 min read

LLM
LLM Serving
Evaluation
Cost Optimization

Table of Contents:

What Is Teacher-Student Distillation for LLMs?
When Distillation Beats Routing and Fine-Tuning
Step 1: Define the Task and Success Criteria
Step 2: Generate Training Data from the Teacher
Step 3: Curate and Filter the Dataset
Step 4: Choose and Prepare the Student Model
Step 5: Run the Training Loop
Step 6: Evaluate the Student Model
Step 7: Deploy and Monitor in Production
Advanced Techniques
Frequently Asked Questions

What Is Teacher-Student Distillation for LLMs?

Short answer: Teacher-student distillation for LLMs is the process of training a small "student" model to replicate a large "teacher" model's behavior on a specific task — producing a cheaper, faster model that matches the teacher's quality on that task.

The teacher (GPT-4o, Claude Sonnet, a 70B open model) generates high-quality outputs. The student (Llama 3.1 8B, Mistral 7B, Phi-3) learns from those outputs via fine-tuning. The result: a model that costs 10–50x less per token and runs on a single GPU — for your specific task.

We've used teacher-student distillation at HinterBuild to replace frontier API calls with self-hosted 8B models for classification, extraction, and domain-specific Q&A — cutting inference costs by 85%+ while maintaining 95%+ of teacher quality.

Key Takeaways:

Teacher-student distillation transfers capability from a large model to a small one for a specific task

Data quality from the teacher matters more than dataset size — 5K excellent examples beat 50K noisy ones

LoRA fine-tuning on an 8B student takes 2–8 hours on a single A100

Always evaluate student vs teacher on held-out data before deploying

Distillation + LLM routing gives maximum cost optimization

This is a complete step-by-step tutorial: data generation, training loop, evaluation, and production deployment.

When Distillation Beats Routing and Fine-Tuning

Three approaches to cheaper LLM inference:

Approach	Cost reduction	Quality retention	Effort
LLM routing	40–70%	90–98%	Low (days)
Fine-tuning (human labels)	50–80%	85–95%	Medium (weeks)
Teacher-student distillation	80–95%	90–98%	Medium (weeks)

Choose Distillation When:

You have a high-volume, well-defined task (classification, extraction, formatting)
The task requires domain knowledge a base small model lacks
You need self-hosted inference for privacy or latency
LLM routing to cheap models isn't hitting quality thresholds
You have budget for a one-time training effort but want ongoing cost savings

Skip Distillation When:

Task is too broad or open-ended (general chatbot)
Traffic is too low to justify training (<10K requests/month)
Quality requirements demand frontier model reasoning
You can hit quality targets with routing alone

Distillation works best as a complement to routing: distill the high-volume task, route edge cases to the teacher.

Step 1: Define the Task and Success Criteria

Before generating a single training example, nail down what "good" means.

Task Definition Template

yaml

task:
  name: "support_ticket_classification"
  description: "Classify customer support tickets into intent categories"
  input: "Raw ticket text (subject + body)"
  output: "JSON with intent, urgency, confidence, suggested_team"

  teacher_model: "gpt-4o"
  student_model: "meta-llama/Llama-3.1-8B-Instruct"

  success_criteria:
    intent_accuracy: ">= 0.95"
    urgency_accuracy: ">= 0.90"
    json_validity: ">= 0.99"
    latency_p99: "<= 500ms"
    cost_per_1k_requests: "<= $0.05"

Output Schema

Define the exact output format. Use constrained JSON decoding when generating teacher data:

python

from pydantic import BaseModel
from typing import Literal

class TicketClassification(BaseModel):
    intent: Literal[
        "billing", "shipping", "returns", "product_question",
        "account", "technical", "other"
    ]
    urgency: Literal["low", "medium", "high", "critical"]
    confidence: float
    suggested_team: Literal["tier1", "tier2", "billing", "engineering"]
    summary: str

The student must produce this exact schema. Define it once, use it for teacher generation, training data, and evaluation.

Step 2: Generate Training Data from the Teacher

Teacher-student distillation quality is bounded by teacher data quality. This is the most important step.

Data Generation Pipeline

python

import asyncio
import json
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

TEACHER_MODEL = "gpt-4o"
SYSTEM_PROMPT = """You are an expert support ticket classifier.
Analyze the ticket and classify it according to the schema.
Be precise with intent categories. When uncertain, use 'other' with lower confidence."""

async def generate_teacher_response(ticket: str) -> dict:
    response = client.beta.chat.completions.parse(
        model=TEACHER_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this ticket:\n\n{ticket}"},
        ],
        response_format=TicketClassification,
        temperature=0.1,  # low temperature for consistent labels
    )
    return response.choices[0].message.parsed.model_dump()

async def generate_dataset(tickets: list[str], output_path: str):
    results = []
    semaphore = asyncio.Semaphore(10)  # rate limit

    async def process(ticket: str, idx: int):
        async with semaphore:
            try:
                label = await generate_teacher_response(ticket)
                results.append({
                    "id": idx,
                    "input": ticket,
                    "output": label,
                    "teacher_model": TEACHER_MODEL,
                })
            except Exception as e:
                print(f"Failed on ticket {idx}: {e}")

    tasks = [process(ticket, i) for i, ticket in enumerate(tickets)]
    await asyncio.gather(*tasks)

    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)

    print(f"Generated {len(results)} examples → {output_path}")

Where to Get Input Data

Sources for distillation training inputs:

Production logs — Anonymized real queries (best distribution match)
Historical data — Past tickets, documents, queries from your domain
Synthetic generation — Ask the teacher to generate diverse inputs:

python

SYNTHETIC_PROMPT = """Generate 10 diverse customer support tickets for an e-commerce company.
Vary: intent, urgency, tone (angry, polite, confused), length, language complexity.
Format: one ticket per line, separated by ---"""

async def generate_synthetic_inputs(n_batches: int = 100) -> list[str]:
    all_tickets = []
    for _ in range(n_batches):
        response = await client.chat.completions.create(
            model="gpt-4o-mini",  # cheap for input generation
            messages=[{"role": "user", "content": SYNTHETIC_PROMPT}],
            temperature=0.9,  # high diversity
        )
        tickets = response.choices[0].message.content.split("---")
        all_tickets.extend(t.strip() for t in tickets if t.strip())
    return all_tickets

Augmentation — Paraphrase existing inputs for diversity

How Much Data?

Task complexity	Minimum examples	Recommended	Diminishing returns after
Binary classification	500	2,000	5,000
Multi-class (5-10 labels)	1,000	5,000	15,000
Structured extraction	2,000	10,000	30,000
Open-ended generation	5,000	20,000	50,000+

For our support classification task, 5,000 teacher-labeled examples produced a student within 2% of teacher accuracy.

Step 3: Curate and Filter the Dataset

Raw teacher output includes errors. Teacher-student distillation fails when you train on bad labels.

Quality Filtering Pipeline

python

def filter_dataset(examples: list[dict]) -> list[dict]:
    filtered = []

    for ex in examples:
        output = ex["output"]
        if output.get("confidence", 0) < 0.7:
            continue  # teacher wasn't sure, skip

        # Filter 2: Schema validation
        try:
            TicketClassification.model_validate(output)
        except Exception:
            continue

        # Filter 3: Input quality
        if len(ex["input"].strip()) < 10:
            continue  # too short to classify

        # Filter 4: Output quality
        if len(output.get("summary", "")) < 5:
            continue

        filtered.append(ex)

    return filtered

def deduplicate(examples: list[dict], similarity_threshold: float = 0.95) -> list[dict]:
    """Remove near-duplicate inputs using embedding similarity."""
    from openai import OpenAI
    client = OpenAI()

    inputs = [ex["input"] for ex in examples]
    embeddings = client.embeddings.create(
        model="text-embedding-3-small", input=inputs
    ).data

    keep = [True] * len(examples)
    for i in range(len(examples)):
        if not keep[i]:
            continue
        for j in range(i + 1, len(examples)):
            if not keep[j]:
                continue
            sim = cosine_similarity(embeddings[i].embedding, embeddings[j].embedding)
            if sim > similarity_threshold:
                keep[j] = False

    return [ex for ex, k in zip(examples, keep) if k]

Train/Validation/Test Split

python

import random

def split_dataset(examples: list[dict], train=0.8, val=0.1, test=0.1):
    random.shuffle(examples)
    n = len(examples)
    train_end = int(n * train)
    val_end = train_end + int(n * val)

    return (
        examples[:train_end],
        examples[train_end:val_end],
        examples[val_end:],
    )

Critical: Keep test set untouched until final evaluation. Never tune hyperparameters on test data.

Format for Training

Convert to instruction-tuning format:

python

def to_training_format(example: dict) -> dict:
    return {
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this ticket:\n\n{example['input']}"},
            {"role": "assistant", "content": json.dumps(example["output"])},
        ]
    }

training_data = [to_training_format(ex) for ex in train_set]

Step 4: Choose and Prepare the Student Model

The student model choice affects distillation quality, cost, and deployment complexity.

Student Model Selection

Model	Parameters	VRAM (fp16)	Best for
Phi-3 Mini	3.8B	8 GB	Edge deployment, simple tasks
Llama 3.1 8B	8B	16 GB	General-purpose distillation
Mistral 7B	7B	14 GB	European data residency
Llama 3.1 70B	70B	140 GB	Maximum quality retention

For most teacher-student distillation projects, Llama 3.1 8B is the sweet spot: strong base capabilities, single-GPU deployment, large community.

LoRA vs Full Fine-Tuning

Method	Trainable params	VRAM needed	Quality	Training time
LoRA	0.1–1%	16–24 GB	95–98% of full FT	2–8 hours
Full fine-tuning	100%	40–80 GB	Best	8–24 hours
QLoRA	0.1–1% (4-bit)	8–12 GB	93–97% of full FT	3–10 hours

Recommendation: Start with LoRA. Move to full fine-tuning only if LoRA quality isn't sufficient.

Setup Training Environment

bash

pip install torch transformers datasets peft trl accelerate bitsandbytes
pip install wandb  # experiment tracking

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)

lora_config = LoraConfig(
    r=16,               # rank — higher = more capacity, more VRAM
    lora_alpha=32,      # scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~42M / 8B = 0.5%

Step 5: Run the Training Loop

This is the core of teacher-student distillation — training the student on teacher-generated data.

Training with TRL SFTTrainer

python

from trl import SFTTrainer, SFTConfig
from datasets import Dataset

def format_chat(example):
    text = tokenizer.apply_chat_template(
        example["messages"],
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}

train_dataset = Dataset.from_list(training_data).map(format_chat)
val_dataset = Dataset.from_list([to_training_format(ex) for ex in val_set]).map(format_chat)

training_args = SFTConfig(
    output_dir="./distilled-classifier",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    max_seq_length=2048,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=100,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    bf16=True,
    report_to="wandb",
    run_name="ticket-classifier-distillation-v1",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model("./distilled-classifier/final")
tokenizer.save_pretrained("./distilled-classifier/final")

Training Hyperparameters Guide

Parameter	Conservative	Aggressive	Notes
`learning_rate`	1e-4	5e-4	Higher LR risks catastrophic forgetting
`num_epochs`	2	5	Watch val loss — stop when it plateaus
`lora_r`	8	64	Higher rank = more capacity for complex tasks
`batch_size`	8	32	Larger batches stabilize training
`max_seq_length`	1024	4096	Match your longest training example

Monitoring Training

Watch for these signals during teacher-student distillation:

Train loss decreasing, val loss flat — Good, model is learning
Train loss decreasing, val loss increasing — Overfitting; reduce epochs or add data
Both losses flat after epoch 1 — Learning rate too low or data too easy
Train loss spikes — Learning rate too high; reduce by 2–5x

Log to W&B or TensorBoard. Compare eval loss across runs to find optimal hyperparameters.

Multi-Teacher Distillation (Advanced)

For higher quality, generate data from multiple teachers and train on the ensemble:

python

TEACHERS = ["gpt-4o", "claude-sonnet-4-20250514", "gemini-2.0-flash"]

async def multi_teacher_label(ticket: str) -> dict:
    labels = await asyncio.gather(*[
        generate_teacher_response(ticket, model=m) for m in TEACHERS
    ])
    # Use majority vote or highest-confidence label
    return resolve_disagreement(labels)

Multi-teacher labels reduce individual teacher bias and improve student generalization.

Step 6: Evaluate the Student Model

Never deploy a distilled model without rigorous evaluation against the teacher.

Load and Run the Student

python

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype="auto", device_map="auto"
)
student = PeftModel.from_pretrained(base_model, "./distilled-classifier/final")
student.eval()

def student_classify(ticket: str) -> dict:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Classify this ticket:\n\n{ticket}"},
    ]
    input_ids = tokenizer.apply_chat_template(
        messages, return_tensors="pt", add_generation_prompt=True
    ).to(student.device)

    output_ids = student.generate(
        input_ids,
        max_new_tokens=256,
        temperature=0.1,
        do_sample=False,
    )
    response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
    return json.loads(response)

Evaluation Metrics

python

from sklearn.metrics import classification_report, accuracy_score

def evaluate_student(test_set: list[dict]) -> dict:
    predictions = []
    ground_truth = []

    for ex in test_set:
        try:
            pred = student_classify(ex["input"])
            predictions.append(pred)
            ground_truth.append(ex["output"])
        except (json.JSONDecodeError, Exception) as e:
            predictions.append({"intent": "PARSE_ERROR", "urgency": "PARSE_ERROR",
                              "confidence": 0, "suggested_team": "tier1", "summary": ""})

    # Intent accuracy
    intent_acc = accuracy_score(
        [g["intent"] for g in ground_truth],
        [p["intent"] for p in predictions],
    )

    # JSON validity rate
    valid = sum(1 for p in predictions if p.get("intent") != "PARSE_ERROR")
    json_validity = valid / len(predictions)

    # Full classification report
    report = classification_report(
        [g["intent"] for g in ground_truth],
        [p["intent"] for p in predictions],
        output_dict=True,
    )

    return {
        "intent_accuracy": intent_acc,
        "json_validity": json_validity,
        "classification_report": report,
        "n_samples": len(test_set),
    }

Teacher vs Student Comparison

Run the same test set through both models:

python

async def compare_teacher_student(test_set: list[dict]) -> dict:
    teacher_results = []
    student_results = []

    for ex in test_set:
        teacher_label = await generate_teacher_response(ex["input"])
        student_label = student_classify(ex["input"])
        teacher_results.append(teacher_label)
        student_results.append(student_label)

    teacher_acc = accuracy_score(
        [r["intent"] for r in teacher_results],
        [ex["output"]["intent"] for ex in test_set],
    )
    student_acc = accuracy_score(
        [r["intent"] for r in student_results],
        [ex["output"]["intent"] for ex in test_set],
    )

    agreement = accuracy_score(
        [t["intent"] for t in teacher_results],
        [s["intent"] for s in student_results],
    )

    return {
        "teacher_accuracy": teacher_acc,
        "student_accuracy": student_acc,
        "teacher_student_agreement": agreement,
        "quality_retention": student_acc / teacher_acc if teacher_acc > 0 else 0,
    }

Acceptable Quality Thresholds

Metric	Minimum to deploy	Target
Quality retention (student/teacher)	90%	95%+
Teacher-student agreement	85%	92%+
JSON validity	99%	99.9%
Latency vs teacher API	10x faster	20x+ faster

If quality retention is below 90%, try: more training data, higher LoRA rank, multi-teacher labels, or a larger student model.

Step 7: Deploy and Monitor in Production

A distilled model is only valuable if it runs reliably in production.

Serve with vLLM

python

# Start vLLM server with LoRA adapter
# vllm serve meta-llama/Llama-3.1-8B-Instruct \
#   --enable-lora \
#   --lora-modules classifier=./distilled-classifier/final \
#   --max-lora-rank 16

Or integrate into your existing backend API:

python

from fastapi import FastAPI
import httpx

app = FastAPI()
VLLM_URL = "http://localhost:8000/v1"

@app.post("/v1/classify")
async def classify_ticket(ticket: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{VLLM_URL}/chat/completions",
            json={
                "model": "classifier",
                "messages": [
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": f"Classify this ticket:\n\n{ticket}"},
                ],
                "temperature": 0.1,
                "max_tokens": 256,
            },
        )
    result = response.json()
    content = result["choices"][0]["message"]["content"]
    return json.loads(content)

Hybrid Deployment: Student + Teacher Fallback

Combine teacher-student distillation with cascade routing:

python

async def classify_with_fallback(ticket: str) -> dict:
    student_result = student_classify(ticket)

    # Low confidence → escalate to teacher
    if student_result.get("confidence", 0) < 0.75:
        teacher_result = await generate_teacher_response(ticket)
        logger.info("escalated_to_teacher", ticket_hash=hash(ticket))
        return teacher_result

    return student_result

In production, ~85% of requests stay on the student. The remaining 15% escalate to the teacher API. Net cost savings: ~80%.

Production Monitoring

Track with observability and monitoring:

Metric	Alert threshold
`student.confidence_p50`	Drop below 0.80 → data drift
`student.escalation_rate`	Above 25% → retrain needed
`student.json_error_rate`	Above 1% → model degradation
`student.latency_p99`	Above 500ms → scaling issue
`student.intent_distribution`	Shift >10% → new ticket types emerging

Schedule monthly re-evaluation. Retrain when quality retention drops below 90% or escalation rate exceeds 20%.

Deploy on cloud infrastructure with GPU autoscaling and model versioning.

Advanced Techniques

Knowledge Distillation with Logit Matching

Instead of training only on teacher outputs (hard labels), match the student's output distribution to the teacher's logits (soft labels):

python

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):
    # Soft label loss (KL divergence between distributions)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    soft_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (temperature ** 2)

    # Hard label loss (standard cross-entropy)
    hard_loss = F.cross_entropy(student_logits, labels)

    return alpha * soft_loss + (1 - alpha) * hard_loss

Soft labels transfer more knowledge per example — the student learns why the teacher chose an answer, not just what it chose.

Continual Distillation

As production data accumulates, periodically re-distill:

Month 1: Initial distillation (5K examples)
Month 3: Add 2K production examples where student escalated
Month 6: Full re-distillation with 10K examples

This keeps the student current as your data distribution evolves.

Distillation for Tool-Calling Agents

Distill tool calling behavior from a teacher agent:

python

# Teacher generates: (query, tool_call, tool_result, final_answer) trajectories
# Student learns: given query → correct tool_call JSON

class ToolCallExample(BaseModel):
    query: str
    tool_name: str
    tool_arguments: dict
    reasoning: str

This produces a fast, cheap tool-selection model for AI agent systems.

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

What is teacher-student distillation for LLMs?

Teacher-student distillation trains a small LLM (student) to replicate a large LLM's (teacher) behavior on a specific task. The teacher generates training data; the student learns via fine-tuning. The result is a cheaper, faster model for that task.

How is distillation different from fine-tuning?

Fine-tuning uses human-labeled data. Teacher-student distillation uses AI-generated labels from a stronger model. Distillation scales data generation cheaply but inherits teacher biases.

How much training data do I need?

For classification tasks: 2,000–5,000 high-quality teacher-labeled examples. For extraction: 5,000–10,000. Quality matters more than quantity — filter aggressively.

Which student model should I use?

Llama 3.1 8B for most tasks. Phi-3 Mini (3.8B) for edge/simple tasks. Llama 3.1 70B when 8B can't retain enough teacher quality. Start small, scale up only if needed.

How long does distillation training take?

LoRA fine-tuning on 5K examples with Llama 8B: 2–8 hours on a single A100. QLoRA on a consumer GPU (RTX 4090): 4–12 hours.

Can I distill from closed API models?

Yes. Use GPT-4o or Claude as the teacher via API. Generate labeled data, then fine-tune an open-weight student. You can't distill the closed model's weights — only its outputs.

How do I know if distillation worked?

Compare student vs teacher on a held-out test set. Target 90%+ quality retention (student accuracy / teacher accuracy). Also measure JSON validity, latency, and cost per request.

Should I use distillation or LLM routing?

Use both. Distillation for high-volume, well-defined tasks. LLM routing for diverse queries and edge cases. Deploy distilled student as the default route, escalate to teacher on low confidence.

Conclusion

Teacher-student distillation for LLMs turns expensive API calls into cheap self-hosted inference — without sacrificing quality on your specific task. The step-by-step process:

Define the task and success criteria with a strict output schema
Generate 2K–10K teacher-labeled examples with quality filtering
Train the student with LoRA fine-tuning (2–8 hours on one GPU)
Evaluate against the teacher on held-out data (target 90%+ retention)
Deploy with cascade fallback to the teacher for low-confidence cases
Monitor and retrain monthly as data distribution shifts

Distillation + routing + constrained JSON decoding is the cost optimization stack we deploy for production AI agent systems.

At HinterBuild:

AI Agent Development — Distilled models for agent tool calling
RAG & LLM Systems — Domain-specific distilled retrievers and classifiers
Backend API Engineering — Model serving infrastructure
Cloud Infrastructure & DevOps — GPU deployment and autoscaling

Free consultation

Book a free consultation call on teacher-student model distillation

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

AI SystemsSeptember 8, 2026 · 10 min read

Model Distillation for LLMs: How to Build Smaller, Smarter

Learn model distillation for llms through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI SystemsSeptember 14, 2026 · 10 min read

vLLM in Production: PagedAttention, Continuous Batching, and

vLLM in Production guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI SystemsSeptember 14, 2026 · 10 min read

Triton vs vLLM: LLM Serving Framework Comparison for

Triton vs vLLM guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI SystemsSeptember 14, 2026 · 9 min read

LLM Tracing with OpenTelemetry: Complete Observability Guide

Learn llm tracing with opentelemetry through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

What Is Teacher-Student Distillation for LLMs?

When Distillation Beats Routing and Fine-Tuning

Choose Distillation When:

Skip Distillation When:

Step 1: Define the Task and Success Criteria

Task Definition Template

Output Schema

Step 2: Generate Training Data from the Teacher

Data Generation Pipeline

Where to Get Input Data

How Much Data?

Step 3: Curate and Filter the Dataset

Quality Filtering Pipeline

Train/Validation/Test Split

Format for Training

Step 4: Choose and Prepare the Student Model

Student Model Selection

LoRA vs Full Fine-Tuning

Setup Training Environment

Step 5: Run the Training Loop

Training with TRL SFTTrainer

Training Hyperparameters Guide

Monitoring Training

Multi-Teacher Distillation (Advanced)

Step 6: Evaluate the Student Model

Load and Run the Student

Evaluation Metrics

Teacher vs Student Comparison

Acceptable Quality Thresholds

Step 7: Deploy and Monitor in Production

Serve with vLLM

Hybrid Deployment: Student + Teacher Fallback

Production Monitoring

Advanced Techniques

Knowledge Distillation with Logit Matching

Continual Distillation

Distillation for Tool-Calling Agents

Frequently Asked Questions

What is teacher-student distillation for LLMs?

How is distillation different from fine-tuning?

How much training data do I need?

Which student model should I use?

How long does distillation training take?

Can I distill from closed API models?

How do I know if distillation worked?

Should I use distillation or LLM routing?

Conclusion

Related articles

Model Distillation for LLMs: How to Build Smaller, Smarter

vLLM in Production: PagedAttention, Continuous Batching, and

Triton vs vLLM: LLM Serving Framework Comparison for

LLM Tracing with OpenTelemetry: Complete Observability Guide