Measure Whether an Email Course Actually Taught

Measuring whether an email course taught anything is how you stop lying to yourself with open rates. An opened email is evidence of delivery and subject-line luck. It is not evidence that the learner can verify a webhook, sketch a retry path, or stop pasting prod secrets into ChatGPT.

Teaching ops need the same honesty we demand of LLM evaluation: define the outcome, collect artifacts, compare against a baseline, and do not celebrate proxy metrics. This post is the measurement layer that sits on top of review, HITL courses, and source maintenance.

Cadensend does not pretend to be an analytics suite today. It is an MIT-licensed, self-hosted curriculum engine with no hosted signup — not ConvertKit, not a bulk sender. Quizzes and progress tracking are on the roadmap. You can still measure now with reply tasks, staging checks, and a spreadsheet. Clone the engine from GitHub.

Key Takeaways:

• Define one observable outcome per series before you write issue 1.
• Separate delivery metrics (arrived, not duplicated) from learning metrics (task done, explanation in the learner’s words).
• Prefer constructed responses and authentic tasks over multiple choice the model could pass.
• Compare cohorts after a source change; if scores drop, the course went stale or the product moved.
• Sample quality with a human rater using a rubric — the same discipline as HITL approval gates.
• Never optimize the course for the quiz so hard that you stop teaching the job.

Table of Contents:

• Open rate is a delivery metric
• Define the outcome in verbs
• A four-level model that does not require an LMS
• Instruments: tasks, quizzes, diffs, incidents
• Personal vs team vs customer vs onboarding
• Statistics without fake precision
• Cadensend, analytics, and what not to build yet
• Closing the loop: change the course
• Frequently Asked Questions

Open rate is a delivery metric

If nobody opens, you have a subject, preheader, or deliverability problem. Fix that first. Cadensend’s exactly-once design exists so you do not confuse “they ignored me” with “they got issue 3 twice and muted you.”

Once mail arrives, open rate is weakly correlated with learning. People open and archive. Image blockers affect pixel tracking anyway; teaching series should not depend on spyware to know if teaching happened.

Useful delivery metrics:

• Accepted by the provider
• Bounce / complaint (should be near zero on a verified recipient)
• Duplicate suppressed (should be 100% of retries)
• Time from approved → sent (Cadensend targets sub-two-minute p95 after approve)

Those belong in Run Center or your worker logs. They are backend metrics. They are not Kirkpatrick level 2.

If you want a classical frame, the Kirkpatrick model still maps: reaction (was the issue readable), learning (can they perform), behavior (did they do it at work), results (fewer tickets, fewer incidents). Email teams usually stop at reaction because it is in the ESP dashboard. Do not.

Reducing LLM costs also needs outcome metrics: cost per successful learner task, not cost per generated token.

Define the outcome in verbs

Write the series outcome as a sentence you could test on Tuesday:

• “The learner can sketch the auth path and name where tokens are validated.”
• “The new hire can run the staging deploy and the rollback command.”
• “The customer can verify a webhook signature with the public key in docs.”
• “I can explain exactly-once mail in terms of an idempotency key and a pre-send record.”

Bad outcomes: “understands the platform,” “feels confident,” “engaged with content.”

Put the verb in Cadensend’s brief. The planner should map issues to that verb. If issue 4 cannot name how it serves the verb, cut it.

This is the same contract as eval-driven development for models — evaluation-driven development — applied to humans.

A four-level model that does not require an LMS

You do not need all four on a personal system-design series. Levels 2 and a private 3 (you shipped a worker) are enough.

You need 2 and 3 on internal onboarding. You need 2 and 4 on customer curricula (activation).

Level 1 is optional. If you obsess over “delight,” you will get fluff. Review already covers email fitness before send.

Instruments: tasks, quizzes, diffs, incidents

Authentic task (best). Do the job in a safe environment. Staging webhook. Rollback. Draw the diagram from memory and photograph it. Tasks are expensive to score and hard to fake.

Constructed response. “Name the three hops. Where does auth happen?” Score with a rubric: 0 missing, 1 partial, 2 correct and matches the source. Humans score until you have examples; models can pre-flag, not certify. Hallucination in a grader is a known failure — keep the rubric and the source span next to the answer.

Multiple choice (weakest). Easy to auto-grade, easy to teach to. Use only for diagnostic smoke tests, never as the only evidence.

Errata comprehension. After a source change, ask what changed. If they still quote the old timeout, the errata issue failed.

Incident and ticket tags. For onboarding and customer series, tag tickets taught-in-issue-4 vs never-covered. That is how you find coverage holes the planner missed.

Diagram redraw. If the PNG was the lesson, they should reproduce the claim in words. See diagrams in email.

Do not put tracking pixels in teaching mail as your primary instrument. Corporate clients block them. They also train you to write clickbait subjects.

Example rubric for “where does auth happen?” (0–2):

Two raters should agree within one point. If they do not, the issue was ambiguous — fix the lesson, not the learners. Store issue_version next to the score so a later regen cannot pollute the dataset. That is the same versioning instinct as HITL written courses.

When a diagram was the teaching object, require a redraw in words. Color memory (“the blue box”) is not a pass.

Personal vs team vs customer vs onboarding

Personal. You close the laptop and redesign. Pass/fail is private. Still write it down or you will inflate.

Team. Two people, same question, independent answers. If they disagree, the issue was ambiguous. Personal vs team series.

Customer. Activation event in product analytics: first verified webhook. Support tickets on that job should fall. If tickets stay flat, the curriculum is a newsletter. Founders teach customers.

Onboarding. Shadow-duty without self-inflicted pages. A runbook PR. Manager observation in week 3. Internal onboarding.

Same engine, different evidence. Cadensend does not need to be a CDP to make this true.

Statistics without fake precision

A five-person onboarding class is not a clinical trial. Report counts, not p-values.

• 4/5 completed the staging deploy
• 2/5 missed auth placement on the sketch
• Tickets tagged webhook-verify dropped from 9 to 3 the month after the series launched, with the caveat that we also shipped a better error message

Do not A/B subject lines while you still have faithfulness bugs in issue 2. HITL first. HITL written courses.

When you change sources, compare the next cohort to the last. A drop is a stale course or a harder product. Either is actionable.

Inter-rater reliability: if two reviewers score constructed responses differently, fix the rubric. This is the same work as HITL agent review quality.

Nielsen’s old point that people scan rather than read still applies. If your only quiz is buried in paragraph seven, you measured patience.

Cadensend, analytics, and what not to build yet

Cadensend’s current job is plan, ground, write, review, send once. Analytics, quizzes, and branching are Update 3 on the product roadmap. Building a full LMS because you wanted to know if issue 4 taught is how side quests eat the company.

Measure outside:

• Reply-to address that lands in a mailbox you control
• A form with the rubric
• Product events for customer jobs
• On-call shadow checklist

When you do add quizzes in-engine, keep them source-grounded. The question should be answerable from cited chunks, not from model trivia. Otherwise you measure the writer’s hallucinations.

Cadensend is not ConvertKit. Do not import “click heatmap” as a teaching KPI. Self-hosted, MIT, verified-recipient MVP. GitHub.

Use coding assistants to tabulate rubric scores. OpenClaw vs Claude Code. Use the ChatGPT cheat sheet to draft rubrics, then lock them like legal text.

Figure 1. Delivery states are visible. Learning evidence still lives in tasks, quizzes, and product events.

If you need an API that records issue_version + learner_id + rubric_score, that is a small backend service. Contact HinterBuild. Agents: AI agent development, still no unsupervised send.

Closing the loop: change the course

Measurement that does not change the plan is vanity.

• Coverage miss → add an issue or fix the planner brief
• Faithfulness miss → retrieval and review
• Diagram miss → redo the spec, not the prose
• Stale miss → hash ritual
• Task too hard → audience level in the wizard was a lie
• Task too easy → you measured reaction, not learning

Lock issues that pass. Do not regen winners because a model suggested a wittier intro.

Teach yourself system design is the cheapest place to practice this loop: you fail in private, then you fix issue 6.

HinterBuild’s about bias: if we cannot name the outcome, we do not generate twelve issues.

Frequently Asked Questions

How do I measure whether an email course actually taught someone?

Define a verb outcome, collect an artifact (sketch, staging task, short answer), score it with a rubric against sources, and look at real-world behavior for team and customer series. Treat open rate as delivery only.

Is a quiz in issue 5 enough?

It is a start if it is constructed response. It is not enough for onboarding or customer activation. Add an authentic task.

Can I use an LLM to grade answers?

You can pre-score with the source span in context. A human should still audit, especially for safety-critical procedures. Graders hallucinate too.

Does Cadensend include learning analytics?

Not in the MVP. It focuses on grounded generation, review, and exactly-once send. Quizzes and progress are later. You can measure with replies and product events now.

What sample size do I need?

For internal courses, all hires. For customers, the first ten who agree to a task is more informative than 10,000 opens. Do not fake statistical significance.

How do diagrams figure into measurement?

Ask learners to restate the claim without looking. If they remember colors but not where auth happens, the figure failed. See diagrams in email.

Should I A/B test lesson copy?

Not until faithfulness and outcome are stable. Otherwise you will crown a fluent wrong issue.

Conclusion

• Delivery ≠ learning. Track both; do not confuse them.
• Verb outcomes and authentic tasks beat multiple choice and pixels.
• Rubrics + sources keep scores honest; model graders are assistants.
• Close the loop into plan, retrieval, diagrams, and locks.
• Cadensend gives you the teaching control plane; you still owe evidence — MIT, self-hosted, not a bulk analytics ESP.

If you want measurement wired to issue versions and send states, contact HinterBuild. About · LinkedIn.