While building InfoSecQuiz, I have been thinking about what a correct answer actually demonstrates. Remembering the expansion of an acronym is useful knowledge, but it is different from choosing the next step in an investigation.
My view is that an assessment should be clear about which ability it measures. Recall questions have a place. They should not carry claims about practical readiness that the questions cannot support.
Define the ability before writing the question
I would start with a specific objective: recognise a term, interpret evidence, identify a missing control or choose between plausible actions.
Then I would write the question to exercise that ability. Adding a long story to an acronym question does not automatically turn it into a scenario-based assessment. The learner must need the scenario’s facts to answer it.
A useful test is to remove the scenario. If the answer remains obvious from a keyword, the item may still be measuring recall.
Make the evidence matter
Consider this illustrative question:
An administrator removes a malicious file from a website. The file reappears later. The site is still online, and relevant logs are available. Which next action best helps establish how the attacker is retaining access?
The answer options could compare preserving and correlating evidence, repeatedly deleting the file, changing the site’s appearance and assuming a new hosting provider will solve the problem.
That is a starting point, but the distracting options are too weak for an advanced assessment. A stronger version would offer several reasonable investigative steps and supply enough context to justify one priority. Otherwise, learners can answer by eliminating obviously irrelevant choices.
I would also avoid claiming there is one correct operational sequence when the scenario lacks facts about active harm, business impact or available containment options.
Review the explanation as carefully as the answer
The explanation should identify which fact makes the preferred answer appropriate and why the other choices are weaker under the stated conditions.
It should also explain what would change the answer. For example, evidence of ongoing data theft may make immediate containment more urgent than the sequence suggested by a less active scenario.
That teaches conditional reasoning. “Option B is correct because it is best practice” gives the learner little they can transfer to another problem.
Question quality improves through review of the reasoning required and the answers learners actually give.
flowchart TD
A["Define the ability"] --> B["Write question and explanation"]
B --> C{"Scenario facts needed?"}
C -->|No| D["Revise or label as recall"]
D --> B
C -->|Yes| E["Review and trial the item"]
E --> F{"Ambiguity or weak distractors?"}
F -->|Yes| B
F -->|No| G["Use for the stated ability"]
G --> H["Review response patterns"]
H --> B
Examine how people answer
I would review response patterns after questions are used. Which distractors are selected? Does an item consistently confuse otherwise capable learners? Are there comments pointing to ambiguous wording or more than one defensible answer?
Response time can provide context, but I would not equate speed with understanding. Language, accessibility and reading effort affect it too.
I would keep practice and assessment conditions separate. Repeated exposure can improve a score through familiarity, so that result should not be presented as independent evidence of new competence.
Keep the claim proportional to the test
A multiple-choice quiz can examine selected knowledge and decisions. It cannot, on its own, establish that somebody can investigate an unfamiliar live incident, write secure code or communicate under pressure.
For those claims, I would add practical tasks and review the work produced. The quiz can help identify what to practise next. Its value increases when the learner understands exactly what the result does and does not measure.