The User Count We Removed From the Academy Dashboard

Two totals that looked like a funnel

In an academy dashboard review, we found two similarly labelled user counts. One included anonymous or incomplete identity records, including records generated during testing. The other counted identifiable sign-ups.

Operators reasonably read the gap as a learner-funnel stage. But the populations did not support that interpretation. We removed the broader total and kept the clearer registration metric. The difference established no conversion rate.

An identity record is not a learner

The ADL xAPI specification distinguishes the actor from the action. That distinction belongs at the start of LMS dashboard design. A stable academy account key is needed across web and mobile, with documented rules for linking provisional identities.

A proposed dictionary makes the boundaries explicit.

• Sessions count visits, not people.
• Provisional records represent incomplete identity states.
• Anonymous records capture activity not linked to an academy account.
• Registered learners count distinct verified academy accounts.
• Active learners count eligible accounts meeting a named activity rule within a stated period.

The denominator belongs in the definition

GOV.UK’s completion-rate guidance defines completed transactions over started transactions and excludes internal users, test users and robots. The useful lesson is the boundary, not the formula alone.

Every academy KPI needs a documented counted entity, qualifying event, time window, set of exclusions and owner. Counts need a population; rates also need a denominator. A comparison of registered learners and active users is meaningful only once it is clear whether both refer to the same eligible cohort and observation period.

Launch testing needs a separate reporting path

In a small launch population, repeated QA activity can materially change a total. That makes it important to tag test accounts and environments at creation and exclude them through shared reporting logic. A test data KPI should flag tagged identities that leak into adoption reports. The records remain available for debugging, outside production adoption claims.

Every chart makes a promise about its population

Jisc’s learning analytics code of practice calls for clear explanations of metrics and their interpretation. Learning analytics definitions should therefore be accessible from the dashboard. Each card can then be reviewed against concrete checks.

• Trace sample records from source events to the displayed total.
• Check repeated visits, account merges and web-to-mobile identity links.
• Confirm test exclusions apply to cards, exports and historical queries.
• Ask an operator to explain the population without help.

Useful metrics survive a clear definition

A metric should stay when its population supports a useful decision. A sound calculation with an overreaching label needs a new name. When neither the population nor the intended interpretation can be defended, the metric should be removed.

The UK Government’s data quality guidance recommends keeping metadata current and documenting definition changes. Dashboard definitions need versions and effective dates. History should be recalculated under the new rules, or the break marked clearly rather than presented as an uninterrupted trend.

Adoption claims must stop at the evidence

The Government Data Quality Framework calls for plain-language definitions and clear communication of limitations. Academy adoption analytics must distinguish registration, learning activity and fintech feature activation. None should be presented as proof of another.

For App-Learning, metric semantics should be a deliverable alongside charts and embedded learning flows. Removing the ambiguous count made the dashboard smaller and its promise clearer. A useful dashboard does not report the largest defensible-looking number. It reports a population the team can explain, reproduce and act on.