About Me

Find out more about me here.

15 October 2012

Teaching Validity and Reliability in the Shooting Gallery

Validity and reliability in research design can be difficult concepts to teach. I've found that a good allegory involves the shooting range. Even if your students aren't experienced with guns or bows (Lord knows I'm not!), it'll still help them to visualize the ideas.

First, a couple related terms. Conceptualization is the process by which we define abstract concepts. In shooting terms, this is how we place the bull's-eye. Operationalization is the process by which we turn those concepts into measures (i.e. how we write the questions or define the variable). In shooting terms, these are the arrows or the bullets.

Validity answers the question, "Am I measuring what I think I am?" In shooting terms, this is "accuracy." My shots may or may not be loosely clustered, but they're all relatively close to the bull's-eye. Reliability answer the question, "Am I consistent?" In shooting terms, this is "precision." My shots may or may not be relatively close to the bull's-eye, but they're tightly clustered. This leads us to four possible outcomes as illustrated below.



The worst-case scenario, when we have low validity and low reliability (lower left), looks like buckshot, scattered all over the target. We are neither accurate nor precise. We're not measuring what we think we are, and at that even, we're doing it inconsistently.

When we have high validity but low reliability (upper left), our packing may be loose, but the shots are near the bull's-eye. We are accurate but not precise. We're likely measuring what we think we are, just not consistently.

When we have high reliability but low validity (upper right), we may be off of the bull's-eye, but our packing is tight. We are precise but not accurate. We're not measuring what we think we are, but whatever we're measuring, we're doing so consistently.

The best-case scenario, high validity and high reliability (lower right), is when the shots are clustered on the bull's-eye. We are both accurate and precise. In other words, our question/variable consistently measures the intended concept.

--
UPDATE (4:25 PM): Turns out I unwittingly stole this from Neuman (1991:171). Oops.

No comments:

Post a Comment