Measure whether an update video helps users understand a change

Start with what the video is meant to clarify. “Users know where a setting moved” suggests a navigation question. “Administrators recognize a migration deadline” suggests an eligibility and action question. A general view count cannot answer either by itself.

Choose observations that match the goal and record their limits. A person may watch without understanding, understand without clicking, or complete the task after receiving help elsewhere. These are different events.

• Exposure: whether the audience had an opportunity to encounter the explanation.
• Comprehension: whether someone can state the change, scope and next action.
• Task signal: a relevant action or completion event, when available and appropriate.
• Support context: recurring questions before and after publication.
• Causal evidence: a separate design needed to attribute a difference to the video.

For a fictional navigation update, the communication goal is that existing editors can locate the export setting's new path. An illustrative evaluation sheet could ask a small set of reviewers to describe the route after seeing the planned explanation, while logging questions that remain.

This is a diagnostic exercise, not a claim of statistical performance. Do not manufacture sample sizes or improvement percentages. Record the actual participants, procedure and responses if the exercise is conducted.

• Question 1: “Where would you open the export setting now?”
• Question 2: “Which role can change it?”
• Question 3: “Did the export behavior change, or only its location?”
• Support theme to inspect: questions mentioning the old route.
• Task signal, if available: completion of the relevant settings action, interpreted with other release changes in mind.

If reviewers find the route but think the export format changed, the navigation lesson may be clear while the scope is misleading. Repair the specific implication rather than celebrate an overall positive reaction. If they remember a feature name but cannot locate it, add orientation context.

Support questions can reveal gaps, but changes in volume may reflect audience size, other documentation or release incidents. Preserve those competing explanations in the analysis.

A useful report states the goal, method, observations, uncertainty and resulting edit. For example: “In the recorded review, several participants confused the moved setting with a new format; we added an unchanged-behavior line.” Use that wording only when the observation occurred.

Avoid saying a video reduced support demand or increased adoption solely because those measures changed after publication. When a stronger causal claim matters, involve the person responsible for the evaluation design before collecting or interpreting the data.