The Quiz Two Weeks Later Changes the Learning Verdict

Key takeaways

Delayed retention is a stronger learning outcome than completion or immediate recall alone.
Comparable content and assessment conditions make differences between variants interpretable.
Random assignment and preregistration separate the planned test from selective reporting.
Abandonment, frustration and time cost belong beside the retention score.
Missing responses and uncertainty can change the learning verdict.

Immediate scores can reverse the verdict

Completion records that someone finished a course. Click-through records activity. Neither establishes what an employee remembers when a suspicious transaction needs escalation two weeks later.

In Roediger and Karpicke’s retrieval-practice experiment (psychologicalscience.org), restudying produced better recall after five minutes, but testing produced better retention after longer delays. An immediate quiz can therefore favour the wrong design for durable learning. LMS learning outcomes need a longer measurement window.

Spaced retrieval becomes a testable product decision

The proposed learning effectiveness A/B test compares the existing sequence with spaced retrieval microlearning. It is a plan, not a completed experiment, and claims no result. Cepeda and colleagues’ spacing study (laplab.ucsd.edu) provides a rationale for testing distributed practice, not proof that a particular academy schedule will work.

Variant A retains the current lesson and immediate quiz.
Variant B adds brief recall attempts with corrective feedback on days two and seven.

Both variants use the same approved source material and learning objectives. Because B adds practice, the comparison tests the whole package—not spacing alone. A spacing-only test would require matching practice volume.

Assignment and analysis are fixed before exposure

The Australian Centre for Evaluation’s preregistration guidance (evaluation.treasury.gov.au) supports specifying outcomes, sample size, randomisation and analysis before the intervention. For this pilot, eligible learners would be randomly assigned before seeing either variant, with assignment stored against a stable learner ID.

The preregistration should define eligibility, exclusions, scoring, missing-data handling and the stopping rule. Nobody should disappear from the analysis merely because they skipped practice. Required instruction and compliance completion records would remain intact in both groups.

Two learning pathways compare standard instruction with spaced retrieval, ending in a shared day-14 assessment with dropout and effort guardrails. — Test what learners retain after 14 days—not just what they recall immediately.

The delayed quiz measures a defined retention window

The primary outcome would be a 14-day retention assessment, anchored to the initial lesson’s start in both groups. A prespecified response window would accommodate working schedules without allowing unlimited delay.

Both groups would receive the same unseen questions, scoring rubric, reminders and assessment conditions. Questions should sample the agreed knowledge objectives rather than repeat practice wording. The score measures retention at this horizon—not workplace performance, reduced compliance risk or financial return.

Dropout and uncertainty constrain the result

The UK government’s randomised-trial guidance (assets.publishing.service.gov.uk) explains why sample size depends on the effect a trial must detect. This pilot needs a power calculation based on the smallest worthwhile retention gain, plausible score variation and expected nonresponse. A small cohort may establish feasibility without settling effectiveness.

The What Works Clearinghouse’s attrition guidance (ies.ed.gov) identifies missing outcomes as a potential source of bias. Assessment invitations should therefore include learners who abandoned training. Reporting should show response rates by assigned group, retain original assignments and test prespecified assumptions about missing scores. Nonresponse is not automatically zero knowledge.

Guardrails would include abandonment, active learning time and a brief frustration rating, with acceptable limits set beforehand.

Plan a retention pilot with App-Learning.

Discuss

The result changes the sequence, not just the dashboard

The decision report should present the retention difference in percentage points with a confidence interval. A worthwhile gain within guardrail limits supports a staged rollout. Excess frustration calls for redesign. An imprecise result calls for more evidence, not a declaration that the variants are equivalent.

For App-Learning, this defines an academy platform’s wider role. The implementation needs variant assignment, exposure timestamps, delayed assessments and learner-level progress records—not just gamified publishing. The standard is simple: a better learning experience must earn its verdict after the session ends.