Analyzing assessment data: reliability and item analysis

This is a data-interpretation essay wearing the clothes of a concepts essay, and the difference decides the grade. Every question is anchored to supplied numbers — the brief says *use the sample statistics data from the textbooks to respond*, and the first question spells the standard out: is this test reliable, and **what evidence from the statistics supports your answer**. A correct textbook definition of reliability with no number attached has answered half a question. The pattern to repeat nine times is the same: define the concept in a sentence, quote the actual figure from the table, say what that particular figure means for this particular test, and state what an instructor would do about it. Definitions are cheap here; the interpretation is what is being assessed. Write the four moves out as a template once and reuse it, so no answer quietly loses its evidence step.

Note that the questions draw on **two different textbooks**, and mixing them up is a quiet way to lose marks. Questions 1 to 4 use the sample test statistics in Chapter 24 of *Teaching in Nursing: A Guide for Faculty*; questions 5 to 9 use Chapter 11 of *The Nurse Educator's Guide to Assessing Learning Outcomes*. Those are different datasets, so a range or a standard error carried over from the first table into an answer about the second is simply the wrong number. Open both chapters before you start and label your working notes by source. It is also worth doing the arithmetic yourself rather than quoting the textbook's summary sentence, because several of these questions ask what the statistic *provides* rather than what it *is*. Note which chapter each figure came from as you transcribe it, not afterwards from memory.

Standard error of measurement is the question most often answered without content, and it has a trap in its wording. The SEM describes the precision of an *individual student's* score — the band within which their true score probably falls — rather than the quality of the test as a whole, which is what students tend to write. The brief then asks whether this test has a small or large SEM, and small or large is meaningless in the abstract: an SEM of 3 is tight on a 100-point exam and disastrous on a 20-point quiz. Judge it against the score scale and the spread of the raw scores, say what you are comparing it to, and then draw the practical conclusion — most usefully, what it means for students clustered near a pass mark. That last point is usually the most useful sentence in the whole essay for a working instructor.

The item-analysis questions are where you can show genuine judgement, because the statistics are advisory rather than decisive. Difficulty index and discrimination index between them flag items worth examining, and the two interact: research on health-professions examinations finds a negative correlation between average difficulty and average discrimination, so easier items tend to discriminate less well. But thresholds do not settle anything on their own. A six-year analysis of over fourteen thousand items found that standard psychometric cut-offs identified about 85.7% of flawed questions and missed 14.3% — with multiple correct answers and incorrect answer keys the commonest defects. The right answer to *what does an instructor do next* is therefore expert review, prompted by the statistic rather than replaced by it. Naming the two commonest defects also tells the instructor what to look for during that review.

Two practical constraints shape the writing. Nine questions inside 1,000–1,250 words leaves roughly 110 to 140 words each, which is enough for the four-move pattern above and nothing more — so there is no room for an introduction that explains why assessment matters, and none for a conclusion that summarises what you just said. Second, keep the instructor's perspective throughout: three of the questions ask explicitly how an instructor would *use* the information, and that is the difference between describing a statistic and demonstrating you could act on one. Answer in the order asked, signpost each question, and let the numbers carry the argument. Numbering the answers one to nine is the simplest way to make coverage visible, and it costs nine words in total rather than a paragraph of signposting prose.